Jul 13 20:53:44.146030 ip-10-0-130-205 systemd[1]: Starting Kubernetes Kubelet... Jul 13 20:53:44.586327 ip-10-0-130-205 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 20:53:44.586327 ip-10-0-130-205 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 20:53:44.586327 ip-10-0-130-205 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 20:53:44.586327 ip-10-0-130-205 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 20:53:44.586327 ip-10-0-130-205 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 20:53:44.588438 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.588329 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 20:53:44.595227 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595213 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.595227 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595227 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595231 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595234 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595238 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595241 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595243 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595246 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595249 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595252 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595255 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595257 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595260 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595263 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595265 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595271 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595274 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595277 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595279 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595282 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595285 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595287 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.595290 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595291 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595294 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595297 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595300 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595302 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595305 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595308 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595312 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595316 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595319 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595322 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595325 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595328 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595331 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595334 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595337 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595340 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595342 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595345 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.595814 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595347 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595350 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595353 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595356 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595359 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595362 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595365 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595368 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595371 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595373 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595376 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595378 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595380 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595383 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595385 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595389 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595392 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595395 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595398 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595400 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.596262 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595403 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595405 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595408 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595410 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595413 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595416 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595419 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595421 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595423 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595426 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595430 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595435 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595438 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595441 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595443 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595446 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595448 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595451 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595453 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.596766 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595456 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595459 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595461 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595464 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595466 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595469 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595471 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595474 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595477 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595479 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595482 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595484 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595487 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595490 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595492 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595495 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595497 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595499 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.595502 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596201 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.597224 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596208 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596211 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596214 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596218 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596221 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596225 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596228 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596231 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596234 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596237 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596240 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596243 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596245 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596248 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596251 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596254 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596256 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596259 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596261 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596264 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.597729 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596267 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596269 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596272 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596275 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596277 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596280 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596282 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596286 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596288 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596290 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596293 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596295 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596299 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596303 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596306 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596309 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596312 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596315 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596318 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.598241 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596322 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596326 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596329 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596331 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596334 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596337 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596340 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596343 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596346 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596348 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596351 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596354 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596358 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596361 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596365 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596368 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596370 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596373 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596376 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596379 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.598721 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596381 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596385 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596387 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596390 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596392 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596395 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596397 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596400 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596403 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596405 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596407 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596411 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596414 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596417 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596419 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596422 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596424 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596427 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596429 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.599216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596432 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596434 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596437 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596439 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596442 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596444 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596447 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596449 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596452 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596454 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596457 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596459 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596462 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596464 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596467 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596470 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596472 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596475 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596477 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596479 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.599664 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596851 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596857 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596862 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596864 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596868 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596871 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596873 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596876 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596879 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596881 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596884 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596887 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596890 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596892 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596895 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596898 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596900 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596902 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596905 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596907 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596910 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.600157 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596913 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596916 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596918 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596921 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596923 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596926 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596929 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596931 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596934 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596937 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596939 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596942 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596944 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596948 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596951 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596953 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596957 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596959 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596962 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596964 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.600677 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596967 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596969 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596972 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596974 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596977 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596979 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596982 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596984 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596986 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596992 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596995 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.596997 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597000 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597002 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597005 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597008 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597010 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597013 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597015 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597018 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.601232 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597020 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597022 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597025 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597027 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597030 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597034 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597037 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597039 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597042 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597045 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597047 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597050 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597053 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597055 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597057 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597060 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597064 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597068 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597071 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.601777 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597073 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597077 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597080 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597083 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597086 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597088 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597091 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597094 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597099 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597102 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597104 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597107 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597109 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597112 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597116 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597119 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597122 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597124 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597128 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597194 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 20:53:44.602295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597204 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597210 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597214 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597219 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597222 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597226 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597231 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597234 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597237 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597240 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597244 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597246 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597249 2561 flags.go:64] FLAG: --cgroup-root="" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597252 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597255 2561 flags.go:64] FLAG: --client-ca-file="" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597258 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597261 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597264 2561 flags.go:64] FLAG: --cluster-domain="" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597267 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597270 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597274 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597278 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597281 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597284 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 20:53:44.602942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597287 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597290 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597293 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597296 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597299 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597302 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597306 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597310 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597313 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597316 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597318 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597321 2561 flags.go:64] FLAG: --enable-server="true" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597325 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597329 2561 flags.go:64] FLAG: --event-burst="100" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597332 2561 flags.go:64] FLAG: --event-qps="50" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597335 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597338 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597340 2561 flags.go:64] FLAG: --eviction-hard="" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597344 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597347 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597350 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597352 2561 flags.go:64] FLAG: --eviction-soft="" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597355 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597358 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597361 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 20:53:44.603515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597363 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597366 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597369 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597374 2561 flags.go:64] FLAG: --feature-gates="" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597377 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597380 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597383 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597386 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597389 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597392 2561 flags.go:64] FLAG: --help="false" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597395 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597398 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597401 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597406 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597409 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597412 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597415 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597418 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597421 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597424 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597427 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597430 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597433 2561 flags.go:64] FLAG: --kube-reserved="" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597436 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 20:53:44.604167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597439 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597442 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597445 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597447 2561 flags.go:64] FLAG: --lock-file="" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597450 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597453 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597456 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597461 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597463 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597466 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597469 2561 flags.go:64] FLAG: --logging-format="text" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597473 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597476 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597479 2561 flags.go:64] FLAG: --manifest-url="" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597482 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597486 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597489 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597492 2561 flags.go:64] FLAG: --max-pods="110" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597495 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597498 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597501 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597505 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597508 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597510 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597513 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 20:53:44.604833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597520 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597523 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597526 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597529 2561 flags.go:64] FLAG: --pod-cidr="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597532 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597537 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597540 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597543 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597546 2561 flags.go:64] FLAG: --port="10250" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597549 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597552 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d61d963974595c10" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597555 2561 flags.go:64] FLAG: --qos-reserved="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597558 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597561 2561 flags.go:64] FLAG: --register-node="true" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597564 2561 flags.go:64] FLAG: --register-with-taints="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597567 2561 flags.go:64] FLAG: --registry-burst="10" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597570 2561 flags.go:64] FLAG: --registry-qps="5" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597572 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597576 2561 flags.go:64] FLAG: --reserved-memory="" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597580 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597583 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597586 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597589 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597591 2561 flags.go:64] FLAG: --runonce="false" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597594 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 20:53:44.605429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597597 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597600 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597603 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597606 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597610 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597612 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597616 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597618 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597621 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597624 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597627 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597629 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597633 2561 flags.go:64] FLAG: --system-cgroups="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597635 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597640 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597643 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597645 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597649 2561 flags.go:64] FLAG: --tls-min-version="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597652 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597654 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597657 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597660 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597663 2561 flags.go:64] FLAG: --v="2" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597666 2561 flags.go:64] FLAG: --version="false" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597670 2561 flags.go:64] FLAG: --vmodule="" Jul 13 20:53:44.606047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597675 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.597678 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597814 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597818 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597821 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597824 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597827 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597830 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597835 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597838 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597841 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597844 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597847 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597851 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597854 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597857 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597859 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597862 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597865 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597867 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.606653 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597870 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597873 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597875 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597878 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597880 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597883 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597885 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597888 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597890 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597893 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597895 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597898 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597903 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597905 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597908 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597911 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597913 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597916 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597918 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.607158 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597921 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597925 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597928 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597931 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597934 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597937 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597940 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597942 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597945 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597947 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597950 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597952 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597955 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597957 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597960 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597963 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597965 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597967 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597970 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.607623 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597973 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597975 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597978 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597980 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597983 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597985 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597989 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597992 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597995 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.597997 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598000 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598002 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598005 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598007 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598010 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598012 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598015 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598017 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598020 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598022 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.608127 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598025 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598027 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598029 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598032 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598035 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598037 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598040 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598042 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598045 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598048 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598050 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598052 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598055 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598058 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598060 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598063 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598065 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598068 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598072 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598075 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.608602 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598077 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598079 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.598082 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.599084 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.606095 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.606109 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.607049 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.607057 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607119 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607124 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607128 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607131 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607134 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607137 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607140 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.609179 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607143 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607146 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607148 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607151 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607154 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607162 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607165 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607168 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607171 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607173 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607176 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607179 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607181 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607184 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607186 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607189 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607191 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607194 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607197 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607200 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.609555 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607202 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607205 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607208 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607213 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607216 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607219 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607222 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607225 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607228 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607231 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607234 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607237 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607240 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607242 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607245 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607248 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607250 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607253 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607263 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607266 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.610040 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607268 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607271 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607273 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607276 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607279 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607281 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607284 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607286 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607289 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607292 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607294 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607297 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607299 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607302 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607305 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607308 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607311 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607314 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607316 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607319 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.610518 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607321 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607327 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607331 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607334 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607337 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607339 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607342 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607345 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607347 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607350 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607352 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607360 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607363 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607365 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607368 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607370 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607373 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607375 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607378 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607380 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.610997 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607383 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607385 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607388 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607390 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607392 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607395 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607398 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607401 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607404 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607406 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607410 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607413 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.607419 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607572 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607595 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 20:53:44.611468 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607599 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607602 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607605 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607608 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607611 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607613 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607616 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607620 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607623 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607632 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607635 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607638 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607641 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607643 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607646 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607648 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607651 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607653 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607656 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 20:53:44.611851 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607658 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607661 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607663 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607666 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607669 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607672 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607674 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607677 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607679 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607682 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607684 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607687 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607689 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607692 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607694 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607697 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607699 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607702 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607704 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607707 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 20:53:44.612293 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607709 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607711 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607714 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607721 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607724 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607727 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607729 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607732 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607734 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607737 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607739 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607742 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607744 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607747 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607764 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607768 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607770 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607773 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607776 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607778 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 20:53:44.612813 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607781 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607784 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607786 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607789 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607792 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607795 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607798 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607801 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607803 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607806 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607809 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607811 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607814 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607816 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607819 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607821 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607830 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607832 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607835 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607837 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 20:53:44.613547 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607840 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607842 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607845 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607847 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607849 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607852 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607854 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607857 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607859 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607862 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607865 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607867 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607870 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607872 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607874 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607877 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607881 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 20:53:44.614332 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:44.607884 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.607889 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.608825 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.611045 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.612569 2561 server.go:1019] "Starting client certificate rotation" Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.612909 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 20:53:44.614743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.613203 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 20:53:44.618237 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.618218 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 20:53:44.619391 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.619371 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 20:53:44.632945 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.632926 2561 log.go:25] "Validated CRI v1 runtime API" Jul 13 20:53:44.635131 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.635117 2561 log.go:25] "Validated CRI v1 image API" Jul 13 20:53:44.636735 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.636721 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 20:53:44.640158 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.640141 2561 fs.go:135] Filesystem UUIDs: map[281b3a81-5657-47c8-b214-114a7439e8a4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9fbb808b-653e-4263-82f7-b7d3402e5a4e:/dev/nvme0n1p4] Jul 13 20:53:44.640216 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.640158 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 20:53:44.645521 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.645414 2561 manager.go:216] Machine: {Timestamp:2026-07-13 20:53:44.644012675 +0000 UTC m=+0.373072566 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094943 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297554e6fa82e191db08622c889c67 SystemUUID:ec297554-e6fa-82e1-91db-08622c889c67 BootID:d7355b92-2e74-49c5-94cc-b3d9ff10a02d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a5:20:28:04:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a5:20:28:04:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:fd:21:de:1f:a9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 20:53:44.645521 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.645517 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 20:53:44.645629 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.645588 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 20:53:44.647365 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.647351 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 20:53:44.649083 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.649047 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 20:53:44.649218 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.649087 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 20:53:44.649944 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.649931 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 20:53:44.649984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.649947 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 20:53:44.649984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.649961 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 20:53:44.650775 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.650765 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 20:53:44.652720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.652710 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 20:53:44.656159 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656147 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 13 20:53:44.656318 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656308 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 20:53:44.656349 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656335 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 13 20:53:44.656376 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656349 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 20:53:44.656376 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656362 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 20:53:44.656376 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656372 2561 kubelet.go:398] "Adding apiserver pod source" Jul 13 20:53:44.656455 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656381 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 20:53:44.656703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656693 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 20:53:44.656734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.656710 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 20:53:44.664331 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.663736 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 20:53:44.666431 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666415 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 20:53:44.666512 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666453 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 20:53:44.666638 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666625 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 20:53:44.666704 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666644 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 20:53:44.666704 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666653 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 20:53:44.666704 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666662 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 20:53:44.666704 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666675 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 20:53:44.666704 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.666684 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 20:53:44.667334 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667323 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 20:53:44.667382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667336 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 20:53:44.667382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667347 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 20:53:44.667382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667356 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 20:53:44.667382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667368 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 20:53:44.667382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.667380 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 20:53:44.668429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.668417 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 20:53:44.668495 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.668432 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 20:53:44.671526 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.671507 2561 server.go:1268] "Started kubelet" Jul 13 20:53:44.671719 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.671680 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 20:53:44.671804 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.671682 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 20:53:44.671804 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.671769 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 20:53:44.672522 ip-10-0-130-205 systemd[1]: Started Kubernetes Kubelet. Jul 13 20:53:44.672843 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.672825 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 13 20:53:44.673295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.673278 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 20:53:44.676016 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.675986 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 20:53:44.678295 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.678264 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 20:53:44.678388 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.678331 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-205.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 20:53:44.678388 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.678333 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 20:53:44.680845 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.679786 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-205.ec2.internal.18c1f4be165b6cc0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-205.ec2.internal,UID:ip-10-0-130-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-205.ec2.internal,},FirstTimestamp:2026-07-13 20:53:44.671481024 +0000 UTC m=+0.400540915,LastTimestamp:2026-07-13 20:53:44.671481024 +0000 UTC m=+0.400540915,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-205.ec2.internal,}" Jul 13 20:53:44.682659 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.682646 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 20:53:44.682707 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.682659 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 20:53:44.682767 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.682717 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:44.683408 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.683390 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 20:53:44.683408 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.683394 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 20:53:44.683556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.683422 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 20:53:44.683556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.683544 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 13 20:53:44.683556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.683554 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 13 20:53:44.685022 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.684994 2561 factory.go:55] Registering systemd factory Jul 13 20:53:44.685022 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685015 2561 factory.go:223] Registration of the systemd container factory successfully Jul 13 20:53:44.685153 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.685101 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 20:53:44.685219 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.685147 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 20:53:44.685322 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685305 2561 factory.go:153] Registering CRI-O factory Jul 13 20:53:44.685322 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685323 2561 factory.go:223] Registration of the crio container factory successfully Jul 13 20:53:44.685463 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685364 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 20:53:44.685463 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685388 2561 factory.go:103] Registering Raw factory Jul 13 20:53:44.685463 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685400 2561 manager.go:1196] Started watching for new ooms in manager Jul 13 20:53:44.685960 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.685945 2561 manager.go:349] Starting recovery of all containers Jul 13 20:53:44.698297 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.698272 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dnjr" Jul 13 20:53:44.700195 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.700176 2561 manager.go:354] Recovery completed Jul 13 20:53:44.704828 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.704682 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.705609 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.705594 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dnjr" Jul 13 20:53:44.707167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707147 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.707235 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707180 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.707235 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707194 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.707687 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707671 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 20:53:44.707687 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707685 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 20:53:44.707826 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707699 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.707826 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.707703 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 13 20:53:44.708439 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.708424 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.708504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.708448 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.708504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.708461 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.709005 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.708992 2561 policy_none.go:49] "None policy: Start" Jul 13 20:53:44.709047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.709009 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 20:53:44.709047 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.709020 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 20:53:44.709379 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.709316 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-205.ec2.internal.18c1f4be187bebc7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-205.ec2.internal,UID:ip-10-0-130-205.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-205.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-205.ec2.internal,},FirstTimestamp:2026-07-13 20:53:44.707165127 +0000 UTC m=+0.436225022,LastTimestamp:2026-07-13 20:53:44.707165127 +0000 UTC m=+0.436225022,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-205.ec2.internal,}" Jul 13 20:53:44.710024 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.710014 2561 policy_none.go:47] "Start" Jul 13 20:53:44.783059 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.783034 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:44.826377 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.826341 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 20:53:44.827880 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.827863 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 20:53:44.827984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.827904 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 20:53:44.827984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.827930 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 20:53:44.827984 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.827969 2561 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 13 20:53:44.830157 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.830128 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:44.883310 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.883291 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:44.890369 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.890353 2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-205.ec2.internal\" not found" node="ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.891275 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891263 2561 manager.go:337] "Starting Device Plugin manager" Jul 13 20:53:44.891331 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.891303 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 20:53:44.891331 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891315 2561 server.go:87] "Starting device plugin registration server" Jul 13 20:53:44.891618 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891602 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 20:53:44.891716 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891617 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 20:53:44.891795 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891716 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 20:53:44.891867 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891799 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 20:53:44.891867 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.891808 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 20:53:44.892433 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.892401 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 20:53:44.892523 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.892458 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:44.928414 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.928381 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal"] Jul 13 20:53:44.928510 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.928488 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.929456 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.929437 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.929556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.929478 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.929556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.929494 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.930728 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.930713 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.930878 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.930863 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.930929 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.930892 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.931415 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931399 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.931415 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931408 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.931536 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931427 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.931536 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931445 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.931536 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931430 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.931536 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.931517 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.932892 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.932877 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.932942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.932904 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.936772 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.936732 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.936772 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.936772 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.936928 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.936786 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.950283 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.950259 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.ec2.internal\" not found" node="ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.955780 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:44.955763 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.ec2.internal\" not found" node="ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.984470 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.984435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.984545 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.984481 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.984545 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.984496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398ce9638757b84ca9a0be66ab3c6445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.ec2.internal\" (UID: \"398ce9638757b84ca9a0be66ab3c6445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:44.992547 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.992530 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 20:53:44.993267 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.993252 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientMemory" Jul 13 20:53:44.993359 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.993280 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 20:53:44.993359 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.993294 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeHasSufficientPID" Jul 13 20:53:44.993359 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:44.993325 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.004444 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.004424 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.004532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.004448 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-205.ec2.internal\": node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.051232 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.051206 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.085372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.085446 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085377 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.085446 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398ce9638757b84ca9a0be66ab3c6445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.ec2.internal\" (UID: \"398ce9638757b84ca9a0be66ab3c6445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.085446 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085423 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/398ce9638757b84ca9a0be66ab3c6445-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.ec2.internal\" (UID: \"398ce9638757b84ca9a0be66ab3c6445\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.085535 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085448 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.085535 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.085457 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/649debe3994b5a9dbe55df4e44d211df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal\" (UID: \"649debe3994b5a9dbe55df4e44d211df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.151690 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.151623 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.252304 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.252265 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.252304 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.252306 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.259048 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.259031 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.352782 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.352740 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.453242 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.453173 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.498197 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.498171 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:45.553724 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.553688 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-205.ec2.internal\" not found" Jul 13 20:53:45.603944 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.603922 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:45.611521 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.611493 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 20:53:45.611672 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.611645 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 20:53:45.611735 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.611668 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 20:53:45.659456 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.659436 2561 apiserver.go:52] "Watching apiserver" Jul 13 20:53:45.668123 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.668103 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 20:53:45.670079 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.670053 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-m5v6b","openshift-cluster-node-tuning-operator/tuned-46kht","openshift-image-registry/node-ca-8b9q9","openshift-multus/multus-vsrzm","openshift-network-diagnostics/network-check-target-qwwxn","openshift-network-operator/iptables-alerter-72fvp","openshift-ovn-kubernetes/ovnkube-node-n8svj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9","openshift-dns/node-resolver-v5cd5","openshift-multus/multus-additional-cni-plugins-mxnmh","openshift-multus/network-metrics-daemon-97vfm","kube-system/global-pull-secret-syncer-bf8nw"] Jul 13 20:53:45.671590 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.671568 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.672819 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.672794 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.674005 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.673984 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 20:53:45.674144 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.674029 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-slbvw\"" Jul 13 20:53:45.674307 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.674291 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.674355 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.674327 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 20:53:45.675098 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.675076 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.675219 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.675170 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.675280 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.675251 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lp54v\"" Jul 13 20:53:45.675333 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.675302 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 20:53:45.675651 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.675633 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:45.675725 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.675704 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:45.676534 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.676511 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tkc2v\"" Jul 13 20:53:45.676636 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.676519 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.676705 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.676635 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 20:53:45.676799 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.676780 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.676919 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.676903 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 20:53:45.677043 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.677026 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.679171 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.679149 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.679488 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.679468 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.679568 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.679507 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.679568 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.679558 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 20:53:45.679810 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.679791 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dv6bh\"" Jul 13 20:53:45.680424 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.680406 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.680495 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.680476 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:45.681749 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.681734 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 20:53:45.681879 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.681859 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.682148 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.682041 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.682148 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.682119 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.682418 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.682396 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g5b22\"" Jul 13 20:53:45.683030 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.683005 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 20:53:45.683138 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.683019 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.684142 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.684116 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.684337 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.684311 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c9zsw\"" Jul 13 20:53:45.684437 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.684317 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.684437 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.684421 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.688181 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.687819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.688409 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.688258 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 20:53:45.688409 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.688319 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.689282 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.688610 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 20:53:45.689282 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689080 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.689282 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689161 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 20:53:45.689490 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689426 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 20:53:45.689566 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689547 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8rlf\"" Jul 13 20:53:45.689620 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689601 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9798223-c3c2-4b30-8311-e9e6780dde4d-host\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.689670 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689657 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvgf\" (UniqueName: \"kubernetes.io/projected/58293a86-df78-4dae-9e33-71394395d584-kube-api-access-8wvgf\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.689721 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689696 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.689771 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689726 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.689841 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689805 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-etc-tuned\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.689898 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689843 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.689977 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.689956 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-cnibin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.690042 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-systemd-units\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.690096 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd815d21-6e0e-464d-b647-e8e30d093c30-ovn-node-metrics-cert\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.690148 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690120 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.690198 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:45.690248 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-etc-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.690300 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690249 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-bin\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.690565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-socket-dir-parent\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.690642 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690619 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-conf-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.690694 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690666 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysconfig\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.690759 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690741 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-run\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.690815 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690784 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wfl\" (UniqueName: \"kubernetes.io/projected/f165a604-de48-410b-9966-ac097681c65b-kube-api-access-g4wfl\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.690887 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690838 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-multus-certs\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.690887 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690874 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/560fb88a-f2fa-460a-b99b-82cdd163f377-host-slash\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.690982 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690901 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 20:53:45.690982 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690938 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-registration-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.690982 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.690973 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-lib-modules\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.691126 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c610b0f5-60ef-494c-8810-0b9ab64a0df1-agent-certs\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.691126 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691041 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-tmp\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.691126 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691073 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-bin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.691126 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-netns\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.691318 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691176 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-config\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.691369 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-modprobe-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.691369 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691346 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 20:53:45.691486 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-systemd\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.691545 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691507 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9798223-c3c2-4b30-8311-e9e6780dde4d-serviceca\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.691599 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691537 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/560fb88a-f2fa-460a-b99b-82cdd163f377-iptables-alerter-script\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.691686 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-netd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.691759 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691730 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-netns\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.691816 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691788 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z9jmf\"" Jul 13 20:53:45.691816 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.691998 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-kubelet\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.691998 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.691959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9ww\" (UniqueName: \"kubernetes.io/projected/e9798223-c3c2-4b30-8311-e9e6780dde4d-kube-api-access-px9ww\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.692103 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692025 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-hostroot\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.692103 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-script-lib\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.692205 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8tb\" (UniqueName: \"kubernetes.io/projected/bd815d21-6e0e-464d-b647-e8e30d093c30-kube-api-access-tl8tb\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.692252 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-cni-binary-copy\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.692301 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692257 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-device-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.692379 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692357 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-sys\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.692462 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692426 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-var-lib-kubelet\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.692562 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692468 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-system-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.692562 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692510 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-multus\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.692631 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-kubelet\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.692675 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.692743 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-node-log\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.692830 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692806 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-host\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.692890 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692868 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.692964 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.692902 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-k8s-cni-cncf-io\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.693034 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693008 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcvr\" (UniqueName: \"kubernetes.io/projected/560fb88a-f2fa-460a-b99b-82cdd163f377-kube-api-access-7fcvr\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.693134 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-socket-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.693208 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693183 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-sys-fs\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.693361 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxkq\" (UniqueName: \"kubernetes.io/projected/2c838c29-7ff1-4577-ad7e-209822026fb9-kube-api-access-jjxkq\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.693420 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693387 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-ovn\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.693497 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693471 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.693912 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c610b0f5-60ef-494c-8810-0b9ab64a0df1-konnectivity-ca\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.694007 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-conf\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.694007 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-multus-daemon-config\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.694007 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-slash\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694007 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.693987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-systemd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694010 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-var-lib-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694033 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-os-release\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694054 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-etc-kubernetes\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694091 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdvf2\" (UniqueName: \"kubernetes.io/projected/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kube-api-access-zdvf2\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694129 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-log-socket\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694191 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-env-overrides\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.694499 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694217 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" Jul 13 20:53:45.694499 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-kubernetes\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.694586 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.694495 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.694586 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.694546 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:45.695098 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.695081 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal"] Jul 13 20:53:45.695620 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.695602 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 20:53:45.695692 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.695654 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 20:53:45.695917 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.695897 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l6mpf\"" Jul 13 20:53:45.696597 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.696574 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 20:53:45.709931 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.709853 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 20:48:45 +0000 UTC" deadline="2027-12-13 14:05:59.159983909 +0000 UTC" Jul 13 20:53:45.709931 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.709903 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12425h12m13.450085645s" Jul 13 20:53:45.710178 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.710161 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 20:53:45.710250 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.710236 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal"] Jul 13 20:53:45.737269 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.737251 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xrgsm" Jul 13 20:53:45.754134 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.754117 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xrgsm" Jul 13 20:53:45.784261 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.784236 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 20:53:45.795177 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-modprobe-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.795177 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795174 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-systemd\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9798223-c3c2-4b30-8311-e9e6780dde4d-serviceca\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/560fb88a-f2fa-460a-b99b-82cdd163f377-iptables-alerter-script\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795233 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-netd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795248 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-netns\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-kubelet\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795301 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-os-release\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.795352 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795309 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-modprobe-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.795773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795386 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-systemd\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.795773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9ww\" (UniqueName: \"kubernetes.io/projected/e9798223-c3c2-4b30-8311-e9e6780dde4d-kube-api-access-px9ww\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.795773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-hostroot\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.795773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795642 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9798223-c3c2-4b30-8311-e9e6780dde4d-serviceca\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.795773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-script-lib\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795822 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-hostroot\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795861 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8tb\" (UniqueName: \"kubernetes.io/projected/bd815d21-6e0e-464d-b647-e8e30d093c30-kube-api-access-tl8tb\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795899 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-cni-binary-copy\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-device-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.795974 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbeddf55-2119-4a5b-94be-b776003f0a3c-tmp-dir\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.796071 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796054 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-sys\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-var-lib-kubelet\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796085 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/560fb88a-f2fa-460a-b99b-82cdd163f377-iptables-alerter-script\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796110 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-system-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796157 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-multus\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-netns\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-kubelet\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796256 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-netd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-multus\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-node-log\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796374 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-kubelet\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796382 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-system-cni-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-host\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.796432 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-kubelet\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-k8s-cni-cncf-io\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcvr\" (UniqueName: \"kubernetes.io/projected/560fb88a-f2fa-460a-b99b-82cdd163f377-kube-api-access-7fcvr\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-var-lib-kubelet\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796540 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-k8s-cni-cncf-io\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796572 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-socket-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796591 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-sys\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-sys-fs\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxkq\" (UniqueName: \"kubernetes.io/projected/2c838c29-7ff1-4577-ad7e-209822026fb9-kube-api-access-jjxkq\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796665 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-ovn\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796715 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn76c\" (UniqueName: \"kubernetes.io/projected/bbeddf55-2119-4a5b-94be-b776003f0a3c-kube-api-access-xn76c\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c610b0f5-60ef-494c-8810-0b9ab64a0df1-konnectivity-ca\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:45.796905 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649debe3994b5a9dbe55df4e44d211df.slice/crio-1aa31f6ba2b00235c9181f53b75e7d544c34bfc45d7529e96e69622af4996103 WatchSource:0}: Error finding container 1aa31f6ba2b00235c9181f53b75e7d544c34bfc45d7529e96e69622af4996103: Status 404 returned error can't find the container with id 1aa31f6ba2b00235c9181f53b75e7d544c34bfc45d7529e96e69622af4996103 Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796933 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbeddf55-2119-4a5b-94be-b776003f0a3c-hosts-file\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796962 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-cnibin\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796974 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-host\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.797186 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.796997 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bln6c\" (UniqueName: \"kubernetes.io/projected/f89949d5-9982-45e3-be0c-b984d9babe6c-kube-api-access-bln6c\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797056 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-conf\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-script-lib\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-multus-daemon-config\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797114 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-slash\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797142 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-systemd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797200 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-var-lib-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797225 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797257 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-kubelet-config\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797225 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-cni-binary-copy\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797363 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-socket-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:45.797425 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398ce9638757b84ca9a0be66ab3c6445.slice/crio-39b301d3ad21ca2caf21c6a7895b4a8677733f32b769eee1d951a0d907eb4766 WatchSource:0}: Error finding container 39b301d3ad21ca2caf21c6a7895b4a8677733f32b769eee1d951a0d907eb4766: Status 404 returned error can't find the container with id 39b301d3ad21ca2caf21c6a7895b4a8677733f32b769eee1d951a0d907eb4766 Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797516 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-os-release\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797513 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-system-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797561 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-etc-kubernetes\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdvf2\" (UniqueName: \"kubernetes.io/projected/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kube-api-access-zdvf2\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.798059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797646 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-log-socket\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797715 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-env-overrides\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797749 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-dbus\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-kubernetes\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-var-lib-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797827 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9798223-c3c2-4b30-8311-e9e6780dde4d-host\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797873 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9798223-c3c2-4b30-8311-e9e6780dde4d-host\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797875 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvgf\" (UniqueName: \"kubernetes.io/projected/58293a86-df78-4dae-9e33-71394395d584-kube-api-access-8wvgf\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797925 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.797582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-node-log\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798107 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798139 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-etc-tuned\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798180 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-systemd\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798157 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/58293a86-df78-4dae-9e33-71394395d584-multus-daemon-config\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.798907 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798302 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-slash\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798355 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-conf\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798420 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-cnibin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-os-release\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-systemd-units\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798489 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd815d21-6e0e-464d-b647-e8e30d093c30-ovn-node-metrics-cert\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798497 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-sys-fs\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798520 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798549 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798561 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798582 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-cni-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-etc-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798645 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-etc-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798846 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysctl-d\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798853 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-device-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.799703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.798997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-kubernetes\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799084 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-systemd-units\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799097 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-cnibin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799240 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c610b0f5-60ef-494c-8810-0b9ab64a0df1-konnectivity-ca\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799307 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-etc-kubernetes\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799310 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-ovn\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799354 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-env-overrides\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799385 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-log-socket\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799363 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-bin\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799421 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-cni-bin\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-socket-dir-parent\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799454 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-run-openvswitch\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799470 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-conf-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.799489 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799547 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-socket-dir-parent\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799542 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-ovn-kubernetes\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799586 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-multus-conf-dir\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799624 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysconfig\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799654 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-run\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.800792 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799680 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wfl\" (UniqueName: \"kubernetes.io/projected/f165a604-de48-410b-9966-ac097681c65b-kube-api-access-g4wfl\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.799707 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:46.299681146 +0000 UTC m=+2.028741064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799758 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-multus-certs\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799852 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/560fb88a-f2fa-460a-b99b-82cdd163f377-host-slash\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-registration-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799906 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-etc-sysconfig\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799925 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-run\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799928 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-lib-modules\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-run-multus-certs\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.799985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c610b0f5-60ef-494c-8810-0b9ab64a0df1-agent-certs\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800015 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/560fb88a-f2fa-460a-b99b-82cdd163f377-host-slash\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800023 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f165a604-de48-410b-9966-ac097681c65b-lib-modules\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-tmp\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-bin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b235edd3-5aec-4bfc-a20d-53ed84f563a1-registration-dir\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800104 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-netns\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800130 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-config\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.801720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/58293a86-df78-4dae-9e33-71394395d584-host-var-lib-cni-bin\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.802654 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd815d21-6e0e-464d-b647-e8e30d093c30-host-run-netns\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.802654 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.800986 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd815d21-6e0e-464d-b647-e8e30d093c30-ovnkube-config\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.802761 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.802699 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-tmp\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.802878 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.802756 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f165a604-de48-410b-9966-ac097681c65b-etc-tuned\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.802878 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.802794 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd815d21-6e0e-464d-b647-e8e30d093c30-ovn-node-metrics-cert\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.803773 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.803665 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c610b0f5-60ef-494c-8810-0b9ab64a0df1-agent-certs\") pod \"konnectivity-agent-m5v6b\" (UID: \"c610b0f5-60ef-494c-8810-0b9ab64a0df1\") " pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:45.804403 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.804385 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 20:53:45.805037 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.805011 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9ww\" (UniqueName: \"kubernetes.io/projected/e9798223-c3c2-4b30-8311-e9e6780dde4d-kube-api-access-px9ww\") pod \"node-ca-8b9q9\" (UID: \"e9798223-c3c2-4b30-8311-e9e6780dde4d\") " pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:45.808175 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.808137 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:53:45.808175 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.808159 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:53:45.808175 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.808172 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:45.808422 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.808227 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:53:46.308211005 +0000 UTC m=+2.037270884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:45.810368 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.810345 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvgf\" (UniqueName: \"kubernetes.io/projected/58293a86-df78-4dae-9e33-71394395d584-kube-api-access-8wvgf\") pod \"multus-vsrzm\" (UID: \"58293a86-df78-4dae-9e33-71394395d584\") " pod="openshift-multus/multus-vsrzm" Jul 13 20:53:45.811114 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.811090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcvr\" (UniqueName: \"kubernetes.io/projected/560fb88a-f2fa-460a-b99b-82cdd163f377-kube-api-access-7fcvr\") pod \"iptables-alerter-72fvp\" (UID: \"560fb88a-f2fa-460a-b99b-82cdd163f377\") " pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:45.811546 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.811516 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdvf2\" (UniqueName: \"kubernetes.io/projected/b235edd3-5aec-4bfc-a20d-53ed84f563a1-kube-api-access-zdvf2\") pod \"aws-ebs-csi-driver-node-wb8x9\" (UID: \"b235edd3-5aec-4bfc-a20d-53ed84f563a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:45.811647 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.811577 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxkq\" (UniqueName: \"kubernetes.io/projected/2c838c29-7ff1-4577-ad7e-209822026fb9-kube-api-access-jjxkq\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:45.812188 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.812165 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wfl\" (UniqueName: \"kubernetes.io/projected/f165a604-de48-410b-9966-ac097681c65b-kube-api-access-g4wfl\") pod \"tuned-46kht\" (UID: \"f165a604-de48-410b-9966-ac097681c65b\") " pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.812763 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.812738 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8tb\" (UniqueName: \"kubernetes.io/projected/bd815d21-6e0e-464d-b647-e8e30d093c30-kube-api-access-tl8tb\") pod \"ovnkube-node-n8svj\" (UID: \"bd815d21-6e0e-464d-b647-e8e30d093c30\") " pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.814418 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.814393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-46kht" Jul 13 20:53:45.820439 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.820417 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:53:45.823180 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:45.823158 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf165a604_de48_410b_9966_ac097681c65b.slice/crio-53cc6768de455ecd0d2d58d5c958feb3c7faa3af385816e5b6ba7e1742d4a264 WatchSource:0}: Error finding container 53cc6768de455ecd0d2d58d5c958feb3c7faa3af385816e5b6ba7e1742d4a264: Status 404 returned error can't find the container with id 53cc6768de455ecd0d2d58d5c958feb3c7faa3af385816e5b6ba7e1742d4a264 Jul 13 20:53:45.828088 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:45.828065 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd815d21_6e0e_464d_b647_e8e30d093c30.slice/crio-3540d15e6d3faa8aa3ab9e4f89669023bf63d0b4c372d7620eb6f6c8fd83b356 WatchSource:0}: Error finding container 3540d15e6d3faa8aa3ab9e4f89669023bf63d0b4c372d7620eb6f6c8fd83b356: Status 404 returned error can't find the container with id 3540d15e6d3faa8aa3ab9e4f89669023bf63d0b4c372d7620eb6f6c8fd83b356 Jul 13 20:53:45.834772 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.834731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46kht" event={"ID":"f165a604-de48-410b-9966-ac097681c65b","Type":"ContainerStarted","Data":"53cc6768de455ecd0d2d58d5c958feb3c7faa3af385816e5b6ba7e1742d4a264"} Jul 13 20:53:45.836026 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.835999 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" event={"ID":"398ce9638757b84ca9a0be66ab3c6445","Type":"ContainerStarted","Data":"39b301d3ad21ca2caf21c6a7895b4a8677733f32b769eee1d951a0d907eb4766"} Jul 13 20:53:45.837059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.837039 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" event={"ID":"649debe3994b5a9dbe55df4e44d211df","Type":"ContainerStarted","Data":"1aa31f6ba2b00235c9181f53b75e7d544c34bfc45d7529e96e69622af4996103"} Jul 13 20:53:45.838231 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.838210 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"3540d15e6d3faa8aa3ab9e4f89669023bf63d0b4c372d7620eb6f6c8fd83b356"} Jul 13 20:53:45.856313 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.856292 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:45.901560 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901528 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-os-release\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.901734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901581 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbeddf55-2119-4a5b-94be-b776003f0a3c-tmp-dir\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.901734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901601 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.901734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901631 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-system-cni-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.901734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn76c\" (UniqueName: \"kubernetes.io/projected/bbeddf55-2119-4a5b-94be-b776003f0a3c-kube-api-access-xn76c\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.901734 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-os-release\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901806 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbeddf55-2119-4a5b-94be-b776003f0a3c-hosts-file\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901849 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-system-cni-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901864 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-cnibin\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901909 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbeddf55-2119-4a5b-94be-b776003f0a3c-hosts-file\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bln6c\" (UniqueName: \"kubernetes.io/projected/f89949d5-9982-45e3-be0c-b984d9babe6c-kube-api-access-bln6c\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901958 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-cnibin\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.901960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.901814 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902012 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbeddf55-2119-4a5b-94be-b776003f0a3c-tmp-dir\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.902055 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:45.902052 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:46.402028375 +0000 UTC m=+2.131088272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902081 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-kubelet-config\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902138 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f89949d5-9982-45e3-be0c-b984d9babe6c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902143 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-kubelet-config\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902143 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-dbus\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902275 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902354 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03300b5c-abf8-4699-afa0-86ad3a8764c6-dbus\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:45.902605 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902581 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.903019 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902780 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.903019 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.902814 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f89949d5-9982-45e3-be0c-b984d9babe6c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.912795 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.912768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bln6c\" (UniqueName: \"kubernetes.io/projected/f89949d5-9982-45e3-be0c-b984d9babe6c-kube-api-access-bln6c\") pod \"multus-additional-cni-plugins-mxnmh\" (UID: \"f89949d5-9982-45e3-be0c-b984d9babe6c\") " pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:45.912795 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.912781 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn76c\" (UniqueName: \"kubernetes.io/projected/bbeddf55-2119-4a5b-94be-b776003f0a3c-kube-api-access-xn76c\") pod \"node-resolver-v5cd5\" (UID: \"bbeddf55-2119-4a5b-94be-b776003f0a3c\") " pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:45.996065 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:45.995976 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:53:46.004563 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.004532 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc610b0f5_60ef_494c_8810_0b9ab64a0df1.slice/crio-a9c5623adaad1f23814081e72ce9d40497aa9bd44e52eb8e27272dd8fcdd9b40 WatchSource:0}: Error finding container a9c5623adaad1f23814081e72ce9d40497aa9bd44e52eb8e27272dd8fcdd9b40: Status 404 returned error can't find the container with id a9c5623adaad1f23814081e72ce9d40497aa9bd44e52eb8e27272dd8fcdd9b40 Jul 13 20:53:46.023021 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.022994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8b9q9" Jul 13 20:53:46.030100 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.030068 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9798223_c3c2_4b30_8311_e9e6780dde4d.slice/crio-e1e6d18fd86ad297273c39d8e3ad8148763a067cd43560564db52c7dc6ce0115 WatchSource:0}: Error finding container e1e6d18fd86ad297273c39d8e3ad8148763a067cd43560564db52c7dc6ce0115: Status 404 returned error can't find the container with id e1e6d18fd86ad297273c39d8e3ad8148763a067cd43560564db52c7dc6ce0115 Jul 13 20:53:46.052440 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.052411 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsrzm" Jul 13 20:53:46.056492 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.056472 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-72fvp" Jul 13 20:53:46.060671 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.060647 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58293a86_df78_4dae_9e33_71394395d584.slice/crio-85d73c302213f047b0ba9fb3deb377c1013625f8531fbb818ea6df5d58c17669 WatchSource:0}: Error finding container 85d73c302213f047b0ba9fb3deb377c1013625f8531fbb818ea6df5d58c17669: Status 404 returned error can't find the container with id 85d73c302213f047b0ba9fb3deb377c1013625f8531fbb818ea6df5d58c17669 Jul 13 20:53:46.064632 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.064598 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560fb88a_f2fa_460a_b99b_82cdd163f377.slice/crio-36951bc2d99a858d6b2068afb8b52e960d44a2962fa0e596676064db464ce690 WatchSource:0}: Error finding container 36951bc2d99a858d6b2068afb8b52e960d44a2962fa0e596676064db464ce690: Status 404 returned error can't find the container with id 36951bc2d99a858d6b2068afb8b52e960d44a2962fa0e596676064db464ce690 Jul 13 20:53:46.075374 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.075355 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" Jul 13 20:53:46.086795 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.084048 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb235edd3_5aec_4bfc_a20d_53ed84f563a1.slice/crio-8077cd44b6cabaa6d3d2d60abeef5891e43a43485f3354955de2fccc3758b7da WatchSource:0}: Error finding container 8077cd44b6cabaa6d3d2d60abeef5891e43a43485f3354955de2fccc3758b7da: Status 404 returned error can't find the container with id 8077cd44b6cabaa6d3d2d60abeef5891e43a43485f3354955de2fccc3758b7da Jul 13 20:53:46.127691 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.127656 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v5cd5" Jul 13 20:53:46.136281 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.136258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" Jul 13 20:53:46.136537 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.136515 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeddf55_2119_4a5b_94be_b776003f0a3c.slice/crio-025f290287bf3cbba6e36186446be79b54f4448ea9f4154ea417b7d63fbb1f51 WatchSource:0}: Error finding container 025f290287bf3cbba6e36186446be79b54f4448ea9f4154ea417b7d63fbb1f51: Status 404 returned error can't find the container with id 025f290287bf3cbba6e36186446be79b54f4448ea9f4154ea417b7d63fbb1f51 Jul 13 20:53:46.144219 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:53:46.144192 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89949d5_9982_45e3_be0c_b984d9babe6c.slice/crio-223794aad5e2afc9d1998960b944e110bf867e18cf1ab1d2cb8c592768cc45f0 WatchSource:0}: Error finding container 223794aad5e2afc9d1998960b944e110bf867e18cf1ab1d2cb8c592768cc45f0: Status 404 returned error can't find the container with id 223794aad5e2afc9d1998960b944e110bf867e18cf1ab1d2cb8c592768cc45f0 Jul 13 20:53:46.306113 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.306023 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:46.306289 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.306259 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:46.306354 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.306333 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:47.306312312 +0000 UTC m=+3.035372194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:46.407062 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.407019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:46.407263 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.407195 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:46.407263 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407202 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:46.407422 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407288 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:47.407263467 +0000 UTC m=+3.136323353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:46.407422 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407347 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:53:46.407422 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407369 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:53:46.407422 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407383 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:46.407583 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.407439 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:53:47.407418461 +0000 UTC m=+3.136478354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:46.756951 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.756750 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:48:46 +0000 UTC" deadline="2028-04-20 03:28:02.189780534 +0000 UTC" Jul 13 20:53:46.756951 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.756836 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15510h34m15.432949682s" Jul 13 20:53:46.829163 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.829123 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:46.829349 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:46.829266 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:46.863607 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.863563 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5v6b" event={"ID":"c610b0f5-60ef-494c-8810-0b9ab64a0df1","Type":"ContainerStarted","Data":"a9c5623adaad1f23814081e72ce9d40497aa9bd44e52eb8e27272dd8fcdd9b40"} Jul 13 20:53:46.891769 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.891727 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5cd5" event={"ID":"bbeddf55-2119-4a5b-94be-b776003f0a3c","Type":"ContainerStarted","Data":"025f290287bf3cbba6e36186446be79b54f4448ea9f4154ea417b7d63fbb1f51"} Jul 13 20:53:46.908483 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.908442 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" event={"ID":"b235edd3-5aec-4bfc-a20d-53ed84f563a1","Type":"ContainerStarted","Data":"8077cd44b6cabaa6d3d2d60abeef5891e43a43485f3354955de2fccc3758b7da"} Jul 13 20:53:46.919032 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.918942 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerStarted","Data":"223794aad5e2afc9d1998960b944e110bf867e18cf1ab1d2cb8c592768cc45f0"} Jul 13 20:53:46.926855 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.926817 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72fvp" event={"ID":"560fb88a-f2fa-460a-b99b-82cdd163f377","Type":"ContainerStarted","Data":"36951bc2d99a858d6b2068afb8b52e960d44a2962fa0e596676064db464ce690"} Jul 13 20:53:46.941454 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.941380 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsrzm" event={"ID":"58293a86-df78-4dae-9e33-71394395d584","Type":"ContainerStarted","Data":"85d73c302213f047b0ba9fb3deb377c1013625f8531fbb818ea6df5d58c17669"} Jul 13 20:53:46.958099 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.958059 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8b9q9" event={"ID":"e9798223-c3c2-4b30-8311-e9e6780dde4d","Type":"ContainerStarted","Data":"e1e6d18fd86ad297273c39d8e3ad8148763a067cd43560564db52c7dc6ce0115"} Jul 13 20:53:46.986215 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:46.985994 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:47.315749 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.315708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:47.315980 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.315962 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:47.316066 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.316043 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:49.316023009 +0000 UTC m=+5.045082894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.416771 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.416857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.416992 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.417059 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:49.417039981 +0000 UTC m=+5.146099876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.417497 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.417517 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.417531 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:47.417637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.417578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:53:49.417563524 +0000 UTC m=+5.146623409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:47.602054 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.601973 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 20:53:47.758107 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.757982 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:48:46 +0000 UTC" deadline="2028-03-30 09:09:04.971660374 +0000 UTC" Jul 13 20:53:47.758107 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.758025 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15012h15m17.213640989s" Jul 13 20:53:47.830051 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.829240 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:47.830051 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.829404 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:47.830051 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:47.829880 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:47.830051 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:47.829989 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:48.839365 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:48.838869 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:48.839365 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:48.839002 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:49.333984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:49.333895 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:49.334155 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.334070 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:49.334155 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.334129 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:53.334110871 +0000 UTC m=+9.063170778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:49.434912 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:49.434847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:49.435108 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:49.434926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:49.444022 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.443465 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:53:49.444022 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.443494 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:53:49.444022 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.443509 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:49.444022 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.443579 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:53:53.443556613 +0000 UTC m=+9.172616513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:49.444868 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.444489 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:49.444868 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.444566 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:53:53.444543489 +0000 UTC m=+9.173603373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:49.828833 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:49.828530 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:49.829026 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.828936 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:49.829326 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:49.828561 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:49.829497 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:49.829431 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:50.828297 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:50.828255 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:50.828684 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:50.828419 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:51.828882 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:51.828844 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:51.829358 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:51.828991 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:51.829425 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:51.829391 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:51.829573 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:51.829543 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:52.828990 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:52.828429 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:52.828990 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:52.828575 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:53.367519 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:53.367474 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:53.367710 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.367690 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:53.367802 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.367762 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:01.367740234 +0000 UTC m=+17.096800129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:53.468862 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:53.468955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469126 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469148 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469161 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469223 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:54:01.46920296 +0000 UTC m=+17.198262853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469356 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:53.469596 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.469448 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:01.469427767 +0000 UTC m=+17.198487653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:53:53.829617 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:53.829188 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:53.829617 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.829336 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:53.829617 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:53.829432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:53.829617 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:53.829574 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:54.829537 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:54.829505 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:54.829714 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:54.829621 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:55.829061 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:55.828721 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:55.829061 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:55.828875 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:55.829061 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:55.828956 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:55.829061 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:55.829029 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:56.828705 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:56.828673 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:56.829157 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:56.828810 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:57.829064 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:57.829020 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:57.829478 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:57.829042 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:57.829478 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:57.829186 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:53:57.829478 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:57.829241 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:58.828433 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:58.828401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:53:58.828594 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:58.828518 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:53:59.828876 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:59.828841 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:53:59.829358 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:59.828956 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:53:59.829358 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:53:59.829012 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:53:59.829358 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:53:59.829120 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:00.828304 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:00.828268 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:00.828459 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:00.828417 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:01.430041 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:01.430010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:01.430557 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.430172 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:54:01.430557 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.430253 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:17.430233576 +0000 UTC m=+33.159293454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:54:01.531384 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:01.531355 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:01.531528 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:01.531401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:01.531528 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531514 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:54:01.531637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:17.531563366 +0000 UTC m=+33.260623247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:54:01.531637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531578 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:54:01.531637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531606 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:54:01.531637 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531620 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:54:01.531816 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.531678 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:54:17.531660801 +0000 UTC m=+33.260720695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:54:01.829178 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:01.829094 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:01.829178 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:01.829146 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:01.829394 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.829234 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:01.829394 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:01.829369 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:02.829302 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:02.829225 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:02.829775 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:02.829345 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:03.828452 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:03.828418 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:03.828634 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:03.828556 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:03.828634 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:03.828617 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:03.828778 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:03.828737 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:04.830003 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:04.829969 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:04.830488 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:04.830084 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:05.829082 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:05.829048 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:05.829257 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:05.829098 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:05.829257 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:05.829158 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:05.829385 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:05.829277 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:06.828450 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:06.828424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:06.828854 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:06.828533 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:07.829538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:07.829049 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:07.829538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:07.829107 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:07.830370 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:07.829580 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:07.830370 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:07.829641 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:08.022167 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022143 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:54:08.022490 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022467 2561 generic.go:356] "Generic (PLEG): container finished" podID="bd815d21-6e0e-464d-b647-e8e30d093c30" containerID="7ce05d02134a51f9ba1681ae40a477bd52fa1f436598eb3126c76f1a3e2a255c" exitCode=1 Jul 13 20:54:08.022559 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022534 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"9324bb870b0446cf13e384e4174291f688d48fd649d19e522ad6e6800f89119b"} Jul 13 20:54:08.022618 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"f161f634b1acae008f3516797b8cce9cce1a502cc7ac251a43b10359e8a5b671"} Jul 13 20:54:08.022671 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022642 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"d3b1d1b4356b5d181b915afa2867e526d18abe4bcf7a27da2afa026e38b08d18"} Jul 13 20:54:08.022708 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022677 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"3c74f462546bd43c83c15346813a107850dba468904a447a148e2783d9eb91e8"} Jul 13 20:54:08.022708 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022705 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerDied","Data":"7ce05d02134a51f9ba1681ae40a477bd52fa1f436598eb3126c76f1a3e2a255c"} Jul 13 20:54:08.022805 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.022744 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"d3df3ea5c1f135ab9b5194a76a836d497c95eba1118113859e92f777292b5e0e"} Jul 13 20:54:08.023966 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.023941 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" event={"ID":"398ce9638757b84ca9a0be66ab3c6445","Type":"ContainerStarted","Data":"f7d6d405d6e6afc02af1be5b435f47bf9b0981c4afa9407e30c7a731282524d8"} Jul 13 20:54:08.025372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.025353 2561 generic.go:356] "Generic (PLEG): container finished" podID="649debe3994b5a9dbe55df4e44d211df" containerID="0929eedbda8d9e92a65a1c2eb7c91514fbd568a7cee4a6f2337d154b752085d8" exitCode=0 Jul 13 20:54:08.025446 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.025418 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" event={"ID":"649debe3994b5a9dbe55df4e44d211df","Type":"ContainerDied","Data":"0929eedbda8d9e92a65a1c2eb7c91514fbd568a7cee4a6f2337d154b752085d8"} Jul 13 20:54:08.026809 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.026782 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v5cd5" event={"ID":"bbeddf55-2119-4a5b-94be-b776003f0a3c","Type":"ContainerStarted","Data":"c07d1533772e89c0a5bd444e9f627bacb2ad2d486458bb3087a89802060fb84f"} Jul 13 20:54:08.028373 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.028354 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" event={"ID":"b235edd3-5aec-4bfc-a20d-53ed84f563a1","Type":"ContainerStarted","Data":"b9367f818ce9bd78b781195d120ae05600e5dd62091b58c9ce8fd6fe1fcb11e9"} Jul 13 20:54:08.029694 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.029675 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-46kht" event={"ID":"f165a604-de48-410b-9966-ac097681c65b","Type":"ContainerStarted","Data":"4329fde9c09a8c1c8b1789f1d36fa1097f39cf0e7c1d95568ad22daeb0002fca"} Jul 13 20:54:08.031196 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.031170 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="29d19447a3af131c8501a244fccbe8b1b725310d5349ccf6b71e881472c5d9ab" exitCode=0 Jul 13 20:54:08.031275 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.031248 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"29d19447a3af131c8501a244fccbe8b1b725310d5349ccf6b71e881472c5d9ab"} Jul 13 20:54:08.032983 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.032950 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsrzm" event={"ID":"58293a86-df78-4dae-9e33-71394395d584","Type":"ContainerStarted","Data":"f24476bc31688c64c0d6f0c6ac312fa1e72438237e407617f2e882ec13e328c3"} Jul 13 20:54:08.034347 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.034325 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8b9q9" event={"ID":"e9798223-c3c2-4b30-8311-e9e6780dde4d","Type":"ContainerStarted","Data":"fb7954d7bfa3460339975050306800c5115aecf38e6fb4552f78bdc81f0d55a3"} Jul 13 20:54:08.035511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.035495 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5v6b" event={"ID":"c610b0f5-60ef-494c-8810-0b9ab64a0df1","Type":"ContainerStarted","Data":"2bbb426631da6c7d9c921e7345584e9790e886e24e0c9258daa0f34f6f367718"} Jul 13 20:54:08.055291 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.055244 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5v6b" podStartSLOduration=2.283955533 podStartE2EDuration="23.055228754s" totalImagesPullingTime="20.771273221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.055220314 +0000 UTC m=+23.784280215" watchObservedRunningTime="2026-07-13 20:54:08.055228754 +0000 UTC m=+23.784288632" Jul 13 20:54:08.055927 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.055898 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.ec2.internal" podStartSLOduration=22.055888808 podStartE2EDuration="22.055888808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.038976596 +0000 UTC m=+23.768036495" watchObservedRunningTime="2026-07-13 20:54:08.055888808 +0000 UTC m=+23.784948726" Jul 13 20:54:08.070994 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.070958 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-46kht" podStartSLOduration=1.068284897 podStartE2EDuration="22.07094699s" totalImagesPullingTime="21.002662093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.070815433 +0000 UTC m=+23.799875334" watchObservedRunningTime="2026-07-13 20:54:08.07094699 +0000 UTC m=+23.800006890" Jul 13 20:54:08.085230 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.085156 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8b9q9" podStartSLOduration=2.339208339 podStartE2EDuration="23.085140385s" totalImagesPullingTime="20.745932046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.084575621 +0000 UTC m=+23.813635521" watchObservedRunningTime="2026-07-13 20:54:08.085140385 +0000 UTC m=+23.814200288" Jul 13 20:54:08.109330 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.109288 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vsrzm" podStartSLOduration=2.31328116 podStartE2EDuration="23.109272581s" totalImagesPullingTime="20.795991421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.108397636 +0000 UTC m=+23.837457556" watchObservedRunningTime="2026-07-13 20:54:08.109272581 +0000 UTC m=+23.838332489" Jul 13 20:54:08.136048 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.136013 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v5cd5" podStartSLOduration=1.448857804 podStartE2EDuration="22.136001969s" totalImagesPullingTime="20.687144165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:08.129671501 +0000 UTC m=+23.858731401" watchObservedRunningTime="2026-07-13 20:54:08.136001969 +0000 UTC m=+23.865061868" Jul 13 20:54:08.763354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.763323 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 20:54:08.829256 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.829226 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:08.829411 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:08.829348 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:08.904695 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.904550 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T20:54:08.763338121Z","UUID":"86385f22-4bc1-42f9-876f-88977c70ae24","Handler":null,"Name":"","Endpoint":""} Jul 13 20:54:08.906234 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.906211 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 20:54:08.906355 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:08.906252 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 20:54:09.040375 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.040332 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" event={"ID":"b235edd3-5aec-4bfc-a20d-53ed84f563a1","Type":"ContainerStarted","Data":"4ed7d9b27be49c1282a2cab276a08fd89ffdd4f9ceba077161fd2202cc8f4f4c"} Jul 13 20:54:09.042068 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.042041 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-72fvp" event={"ID":"560fb88a-f2fa-460a-b99b-82cdd163f377","Type":"ContainerStarted","Data":"88cbe8cab7c28e5bec57791fe702f2aa9846c9c162056df6c9e16b24702d0917"} Jul 13 20:54:09.044235 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.044186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" event={"ID":"649debe3994b5a9dbe55df4e44d211df","Type":"ContainerStarted","Data":"bf2868260156b9e6147fe84802fcf16ae1ad7f2d9f97d873b23ca7226bc23419"} Jul 13 20:54:09.071969 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.071925 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-72fvp" podStartSLOduration=3.312897481 podStartE2EDuration="24.071908181s" totalImagesPullingTime="20.7590107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:09.057370496 +0000 UTC m=+24.786430418" watchObservedRunningTime="2026-07-13 20:54:09.071908181 +0000 UTC m=+24.800968082" Jul 13 20:54:09.072397 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.072369 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.ec2.internal" podStartSLOduration=23.072359075 podStartE2EDuration="23.072359075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:09.071899078 +0000 UTC m=+24.800958980" watchObservedRunningTime="2026-07-13 20:54:09.072359075 +0000 UTC m=+24.801418975" Jul 13 20:54:09.828658 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.828621 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:09.828658 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:09.828657 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:09.828966 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:09.828739 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:09.828966 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:09.828859 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:10.051464 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:10.051433 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:54:10.052294 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:10.052061 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:54:10.828223 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:10.828193 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:10.828508 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:10.828302 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:11.051521 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.051478 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" event={"ID":"b235edd3-5aec-4bfc-a20d-53ed84f563a1","Type":"ContainerStarted","Data":"b242af7222318777e2642280daebb465e1252024a73e699a551612b279a6a974"} Jul 13 20:54:11.058255 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.058222 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:54:11.058669 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.058632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"ef5db19072cc9f1fa1117434a62479798c00e2ab9a86763dbe0a6f229a5a946a"} Jul 13 20:54:11.058869 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.058842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:54:11.059358 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.059333 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5v6b" Jul 13 20:54:11.069116 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.069070 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wb8x9" podStartSLOduration=2.747740753 podStartE2EDuration="26.069057197s" totalImagesPullingTime="23.321316444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:11.068509421 +0000 UTC m=+26.797569347" watchObservedRunningTime="2026-07-13 20:54:11.069057197 +0000 UTC m=+26.798117101" Jul 13 20:54:11.828970 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.828922 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:11.829147 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:11.828937 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:11.829147 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:11.829073 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:11.829229 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:11.829169 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:12.829178 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:12.828955 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:12.829880 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:12.829234 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:13.066477 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.066448 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:54:13.066898 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.066870 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"c244069396285d2266f392f2a1dba03a91503cb0df5ce820c918e6b53e5b4e47"} Jul 13 20:54:13.067389 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.067175 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:13.067389 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.067263 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:13.067389 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.067299 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:13.067389 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.067305 2561 scope.go:117] "RemoveContainer" containerID="7ce05d02134a51f9ba1681ae40a477bd52fa1f436598eb3126c76f1a3e2a255c" Jul 13 20:54:13.068937 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.068915 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="7207bbfab19120f4081d83c9eed7f874918e18d6f18bf0ef8361dbad15eead81" exitCode=0 Jul 13 20:54:13.069051 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.069013 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"7207bbfab19120f4081d83c9eed7f874918e18d6f18bf0ef8361dbad15eead81"} Jul 13 20:54:13.084059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.084040 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:13.084609 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.084591 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:13.828458 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.828431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:13.828709 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:13.828443 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:13.828709 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:13.828546 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:13.828709 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:13.828610 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:14.075770 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.075730 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:54:14.076273 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.076234 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" event={"ID":"bd815d21-6e0e-464d-b647-e8e30d093c30","Type":"ContainerStarted","Data":"50ed22bbd49cc5689af0ea30af1a8ba09813ab8425ad71de955a3bc3b5691dbb"} Jul 13 20:54:14.103684 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.103640 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" podStartSLOduration=6.802519805 podStartE2EDuration="28.103625365s" totalImagesPullingTime="21.30110556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:14.102068145 +0000 UTC m=+29.831128079" watchObservedRunningTime="2026-07-13 20:54:14.103625365 +0000 UTC m=+29.832685264" Jul 13 20:54:14.312493 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.312470 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bf8nw"] Jul 13 20:54:14.312602 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.312564 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:14.312686 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:14.312663 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:14.315012 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.314931 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-97vfm"] Jul 13 20:54:14.315120 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.315019 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:14.315120 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:14.315108 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:14.326314 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.326295 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qwwxn"] Jul 13 20:54:14.326406 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:14.326382 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:14.326461 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:14.326445 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:15.080215 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:15.080177 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="16d2ab522b23809f6ec2b3b683608135081d7b71ea228bac752b92dc32f95006" exitCode=0 Jul 13 20:54:15.080663 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:15.080263 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"16d2ab522b23809f6ec2b3b683608135081d7b71ea228bac752b92dc32f95006"} Jul 13 20:54:15.828409 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:15.828230 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:15.828586 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:15.828268 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:15.828586 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:15.828509 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:15.828586 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:15.828269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:15.828586 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:15.828547 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:15.828821 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:15.828624 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:17.087504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.087424 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="cf38583f0c294b61afa6b1644454d6e1ba7f75567053cc9247c9a55c721f8e70" exitCode=0 Jul 13 20:54:17.087504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.087487 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"cf38583f0c294b61afa6b1644454d6e1ba7f75567053cc9247c9a55c721f8e70"} Jul 13 20:54:17.450553 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.450521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:17.450716 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.450664 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:54:17.450772 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.450722 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:49.450703798 +0000 UTC m=+65.179763689 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 20:54:17.551155 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.551118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:17.551340 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.551181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:17.551340 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551299 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 20:54:17.551340 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551309 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 20:54:17.551340 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551325 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 20:54:17.551340 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551337 2561 projected.go:196] Error preparing data for projected volume kube-api-access-7zm8n for pod openshift-network-diagnostics/network-check-target-qwwxn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:54:17.551598 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551366 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret podName:03300b5c-abf8-4699-afa0-86ad3a8764c6 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:49.551347047 +0000 UTC m=+65.280406928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret") pod "global-pull-secret-syncer-bf8nw" (UID: "03300b5c-abf8-4699-afa0-86ad3a8764c6") : object "kube-system"/"original-pull-secret" not registered Jul 13 20:54:17.551598 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.551389 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n podName:7dd8b245-47d5-4dd2-b945-a7985d96be9b nodeName:}" failed. No retries permitted until 2026-07-13 20:54:49.551373301 +0000 UTC m=+65.280433180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zm8n" (UniqueName: "kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n") pod "network-check-target-qwwxn" (UID: "7dd8b245-47d5-4dd2-b945-a7985d96be9b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 20:54:17.828670 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.828592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:17.828850 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.828592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:17.828850 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.828708 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:17.828850 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:17.828592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:17.828850 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.828825 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:17.829058 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:17.828880 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:19.828439 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:19.828404 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:19.828988 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:19.828411 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:19.828988 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:19.828529 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qwwxn" podUID="7dd8b245-47d5-4dd2-b945-a7985d96be9b" Jul 13 20:54:19.828988 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:19.828412 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:19.828988 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:19.828606 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bf8nw" podUID="03300b5c-abf8-4699-afa0-86ad3a8764c6" Jul 13 20:54:19.828988 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:19.828658 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:54:21.094920 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.094889 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-205.ec2.internal" event="NodeReady" Jul 13 20:54:21.095361 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.095037 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 20:54:21.127836 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.127804 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hx2s6"] Jul 13 20:54:21.142974 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.142955 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:54:21.143113 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.143094 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.145408 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.145388 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 13 20:54:21.145408 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.145394 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4gffg\"" Jul 13 20:54:21.145729 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.145709 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 13 20:54:21.157275 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.157253 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hx2s6"] Jul 13 20:54:21.157369 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.157289 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-252h7"] Jul 13 20:54:21.157369 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.157360 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.159791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.159766 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 20:54:21.159889 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.159826 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 20:54:21.159889 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.159833 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 20:54:21.159889 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.159861 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwbzd\"" Jul 13 20:54:21.167147 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.167047 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 20:54:21.175611 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.175591 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:54:21.175691 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.175618 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bgs6t"] Jul 13 20:54:21.175726 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.175701 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.178034 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.178016 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 20:54:21.178128 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.178033 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f67kb\"" Jul 13 20:54:21.178128 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.178020 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 20:54:21.193641 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.193615 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-252h7"] Jul 13 20:54:21.193726 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.193661 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgs6t"] Jul 13 20:54:21.193726 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.193716 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.195912 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.195897 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q596p\"" Jul 13 20:54:21.195912 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.195905 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 20:54:21.196130 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.196115 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 20:54:21.196177 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.196126 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 20:54:21.277153 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277124 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.277153 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.277354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277287 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277308 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce9d425-a05e-4431-9c38-d928e5728715-config-volume\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.277354 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1833b755-6f8b-4268-b9c9-0471a1755708-cert\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277408 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bvph\" (UniqueName: \"kubernetes.io/projected/1833b755-6f8b-4268-b9c9-0471a1755708-kube-api-access-2bvph\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce9d425-a05e-4431-9c38-d928e5728715-tmp-dir\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277467 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkl7\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277500 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277490 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28sw4\" (UniqueName: \"kubernetes.io/projected/9ce9d425-a05e-4431-9c38-d928e5728715-kube-api-access-28sw4\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.277675 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277675 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277531 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.277675 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.277559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.378381 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.378381 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378385 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.378583 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378403 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.378583 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce9d425-a05e-4431-9c38-d928e5728715-config-volume\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.378583 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.378510 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:21.378583 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378562 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.378583 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.378578 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:21.878559834 +0000 UTC m=+37.607619728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378606 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1833b755-6f8b-4268-b9c9-0471a1755708-cert\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bvph\" (UniqueName: \"kubernetes.io/projected/1833b755-6f8b-4268-b9c9-0471a1755708-kube-api-access-2bvph\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce9d425-a05e-4431-9c38-d928e5728715-tmp-dir\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkl7\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378812 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28sw4\" (UniqueName: \"kubernetes.io/projected/9ce9d425-a05e-4431-9c38-d928e5728715-kube-api-access-28sw4\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.378877 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378841 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378885 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.378977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379003 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379046 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.379109 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379115 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ce9d425-a05e-4431-9c38-d928e5728715-config-volume\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.379123 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:21.379213 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.379194 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:21.879175489 +0000 UTC m=+37.608235368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:21.379646 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379426 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ce9d425-a05e-4431-9c38-d928e5728715-tmp-dir\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.379646 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.379506 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:21.379646 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.379556 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:21.879542186 +0000 UTC m=+37.608602064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:21.379646 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379590 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.379867 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.379832 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.380106 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.380089 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-nginx-conf\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.383061 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.382942 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.383169 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.382966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.383169 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.382997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1833b755-6f8b-4268-b9c9-0471a1755708-cert\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.387766 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.387727 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28sw4\" (UniqueName: \"kubernetes.io/projected/9ce9d425-a05e-4431-9c38-d928e5728715-kube-api-access-28sw4\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.387920 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.387899 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.388718 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.388690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bvph\" (UniqueName: \"kubernetes.io/projected/1833b755-6f8b-4268-b9c9-0471a1755708-kube-api-access-2bvph\") pod \"ingress-canary-bgs6t\" (UID: \"1833b755-6f8b-4268-b9c9-0471a1755708\") " pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.388825 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.388816 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkl7\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.513273 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.513245 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bgs6t" Jul 13 20:54:21.646121 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.646028 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bgs6t"] Jul 13 20:54:21.651762 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:54:21.651725 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1833b755_6f8b_4268_b9c9_0471a1755708.slice/crio-3247ea921571809a94f3c59303a98d961102833deb9213e49cb52fb3a2459f07 WatchSource:0}: Error finding container 3247ea921571809a94f3c59303a98d961102833deb9213e49cb52fb3a2459f07: Status 404 returned error can't find the container with id 3247ea921571809a94f3c59303a98d961102833deb9213e49cb52fb3a2459f07 Jul 13 20:54:21.828268 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.828236 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:21.828268 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.828274 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:21.828452 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.828276 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:21.830896 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.830862 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 20:54:21.830896 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.830863 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 20:54:21.831064 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.830876 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 20:54:21.831064 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.830945 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n7pmh\"" Jul 13 20:54:21.831064 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.830917 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 20:54:21.831212 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.831196 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plnw6\"" Jul 13 20:54:21.883090 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.883073 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:21.883184 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.883109 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:21.883184 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:21.883137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883207 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883223 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883236 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883252 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883261 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:22.883247119 +0000 UTC m=+38.612306996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883273 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:22.883267303 +0000 UTC m=+38.612327181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:21.883306 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:21.883298 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:22.883280418 +0000 UTC m=+38.612340316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:22.102400 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.102374 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="8ad0a4f9f55628846298cb6843248cf04bd434bdb714a7e1f6d4978475f4cf24" exitCode=0 Jul 13 20:54:22.102816 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.102448 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"8ad0a4f9f55628846298cb6843248cf04bd434bdb714a7e1f6d4978475f4cf24"} Jul 13 20:54:22.103620 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.103593 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgs6t" event={"ID":"1833b755-6f8b-4268-b9c9-0471a1755708","Type":"ContainerStarted","Data":"3247ea921571809a94f3c59303a98d961102833deb9213e49cb52fb3a2459f07"} Jul 13 20:54:22.892919 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.892714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:22.893074 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.892955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:22.893074 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.892872 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:22.893074 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:22.892992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:22.893074 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893068 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:24.893045892 +0000 UTC m=+40.622105769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:22.893245 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893102 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:22.893245 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893109 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:22.893245 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893117 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:22.893245 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893163 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:24.893149863 +0000 UTC m=+40.622209752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:22.893245 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:22.893177 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:24.8931715 +0000 UTC m=+40.622231378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:23.109530 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:23.109502 2561 generic.go:356] "Generic (PLEG): container finished" podID="f89949d5-9982-45e3-be0c-b984d9babe6c" containerID="6f94a12b934fce905402c88448cb8b3a10eb01ac964c20f6fa382956f210def2" exitCode=0 Jul 13 20:54:23.109957 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:23.109564 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerDied","Data":"6f94a12b934fce905402c88448cb8b3a10eb01ac964c20f6fa382956f210def2"} Jul 13 20:54:24.115366 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:24.115291 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" event={"ID":"f89949d5-9982-45e3-be0c-b984d9babe6c","Type":"ContainerStarted","Data":"2e13928232c966846214003f952bd45fb9a194fb2ee36979970ea2c16daccdd0"} Jul 13 20:54:24.137909 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:24.137853 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mxnmh" podStartSLOduration=6.434562554 podStartE2EDuration="38.137833555s" totalImagesPullingTime="31.703271001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:24.13727547 +0000 UTC m=+39.866335371" watchObservedRunningTime="2026-07-13 20:54:24.137833555 +0000 UTC m=+39.866893456" Jul 13 20:54:24.912729 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:24.912696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:24.912772 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912805 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:24.912817 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912863 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:28.912847047 +0000 UTC m=+44.641906924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912900 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912909 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:24.912926 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912916 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:24.913187 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912964 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:28.912950284 +0000 UTC m=+44.642010175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:24.913187 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:24.912986 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:28.912975897 +0000 UTC m=+44.642035775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:25.119411 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.119367 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bgs6t" event={"ID":"1833b755-6f8b-4268-b9c9-0471a1755708","Type":"ContainerStarted","Data":"3a29072c3067d46e242cf6d1ea458219186af41f182c3ceced311f79646804c3"} Jul 13 20:54:25.135318 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.135274 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bgs6t" podStartSLOduration=1.461260571 podStartE2EDuration="4.135260442s" totalImagesPullingTime="2.673999871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:25.134204011 +0000 UTC m=+40.863263912" watchObservedRunningTime="2026-07-13 20:54:25.135260442 +0000 UTC m=+40.864320342" Jul 13 20:54:25.166375 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.166309 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k"] Jul 13 20:54:25.168187 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.168168 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.170544 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.170521 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 13 20:54:25.170646 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.170569 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 20:54:25.170646 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.170567 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 20:54:25.170846 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.170829 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 20:54:25.170971 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.170952 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5p45m\"" Jul 13 20:54:25.173717 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.173696 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg"] Jul 13 20:54:25.175603 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.175575 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.186249 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.177881 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 13 20:54:25.189627 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.189607 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k"] Jul 13 20:54:25.191583 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.191508 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg"] Jul 13 20:54:25.315221 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.315185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085539d5-135c-42ef-81a1-98b93d9ea574-tmp\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.315221 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.315222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv2lg\" (UniqueName: \"kubernetes.io/projected/085539d5-135c-42ef-81a1-98b93d9ea574-kube-api-access-fv2lg\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.315448 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.315256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085539d5-135c-42ef-81a1-98b93d9ea574-klusterlet-config\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.315448 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.315347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3530f93f-59b6-47aa-8bac-5831e3ec5a04-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.315525 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.315493 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkds\" (UniqueName: \"kubernetes.io/projected/3530f93f-59b6-47aa-8bac-5831e3ec5a04-kube-api-access-jwkds\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.416442 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3530f93f-59b6-47aa-8bac-5831e3ec5a04-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.416565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416455 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkds\" (UniqueName: \"kubernetes.io/projected/3530f93f-59b6-47aa-8bac-5831e3ec5a04-kube-api-access-jwkds\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.416565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085539d5-135c-42ef-81a1-98b93d9ea574-tmp\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.416565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416491 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv2lg\" (UniqueName: \"kubernetes.io/projected/085539d5-135c-42ef-81a1-98b93d9ea574-kube-api-access-fv2lg\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.416565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416509 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085539d5-135c-42ef-81a1-98b93d9ea574-klusterlet-config\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.416912 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.416887 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/085539d5-135c-42ef-81a1-98b93d9ea574-tmp\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.420014 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.419994 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/085539d5-135c-42ef-81a1-98b93d9ea574-klusterlet-config\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.420224 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.420202 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3530f93f-59b6-47aa-8bac-5831e3ec5a04-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.425890 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.425868 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv2lg\" (UniqueName: \"kubernetes.io/projected/085539d5-135c-42ef-81a1-98b93d9ea574-kube-api-access-fv2lg\") pod \"klusterlet-addon-workmgr-5668c786cf-l9kdg\" (UID: \"085539d5-135c-42ef-81a1-98b93d9ea574\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.426302 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.426280 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkds\" (UniqueName: \"kubernetes.io/projected/3530f93f-59b6-47aa-8bac-5831e3ec5a04-kube-api-access-jwkds\") pod \"managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k\" (UID: \"3530f93f-59b6-47aa-8bac-5831e3ec5a04\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.499654 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.499616 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:54:25.508094 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.508063 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:25.627811 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.627780 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k"] Jul 13 20:54:25.630937 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:54:25.630911 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3530f93f_59b6_47aa_8bac_5831e3ec5a04.slice/crio-cc1df1665da3b46f655907cef79146a5ce8c8b77a8e6acd953a37ed28cc3c7cc WatchSource:0}: Error finding container cc1df1665da3b46f655907cef79146a5ce8c8b77a8e6acd953a37ed28cc3c7cc: Status 404 returned error can't find the container with id cc1df1665da3b46f655907cef79146a5ce8c8b77a8e6acd953a37ed28cc3c7cc Jul 13 20:54:25.641485 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:25.641463 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg"] Jul 13 20:54:25.644084 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:54:25.644061 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085539d5_135c_42ef_81a1_98b93d9ea574.slice/crio-99d9ee9ad34ead561a3538ac09fe2ee1f65bdf883c3e093c418309dcc9712483 WatchSource:0}: Error finding container 99d9ee9ad34ead561a3538ac09fe2ee1f65bdf883c3e093c418309dcc9712483: Status 404 returned error can't find the container with id 99d9ee9ad34ead561a3538ac09fe2ee1f65bdf883c3e093c418309dcc9712483 Jul 13 20:54:26.122911 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:26.122869 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerStarted","Data":"99d9ee9ad34ead561a3538ac09fe2ee1f65bdf883c3e093c418309dcc9712483"} Jul 13 20:54:26.123933 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:26.123907 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerStarted","Data":"cc1df1665da3b46f655907cef79146a5ce8c8b77a8e6acd953a37ed28cc3c7cc"} Jul 13 20:54:28.947544 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:28.947503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:28.947573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947668 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947694 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:28.947669 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947743 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947766 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:36.947733673 +0000 UTC m=+52.676793551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947764 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947865 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:36.947845411 +0000 UTC m=+52.676905291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:28.948003 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:28.947898 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:36.947882958 +0000 UTC m=+52.676942837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:31.138447 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.138412 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerStarted","Data":"5b163b0c29f0964881ee1a486c50bbec7853e4690d0a66c37bc07e3d72e20a1f"} Jul 13 20:54:31.139813 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.139789 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerStarted","Data":"df1a4185d2b421f23769b56618bee07521fb2f8ec1fd56e6eeb4497deb3d6d28"} Jul 13 20:54:31.140057 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.140040 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:31.141564 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.141546 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:54:31.153526 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.153484 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" podStartSLOduration=1.5210219230000002 podStartE2EDuration="6.153469584s" totalImagesPullingTime="4.632447661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:31.15328716 +0000 UTC m=+46.882347060" watchObservedRunningTime="2026-07-13 20:54:31.153469584 +0000 UTC m=+46.882529485" Jul 13 20:54:31.170922 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:31.170890 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" podStartSLOduration=1.535793985 podStartE2EDuration="6.170881819s" totalImagesPullingTime="4.635087834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:31.170770571 +0000 UTC m=+46.899830484" watchObservedRunningTime="2026-07-13 20:54:31.170881819 +0000 UTC m=+46.899941719" Jul 13 20:54:37.012402 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:37.012367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:37.012414 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012511 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012525 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:37.012542 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012570 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:53.012558103 +0000 UTC m=+68.741617981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012512 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012625 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:54:53.012612679 +0000 UTC m=+68.741672557 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012635 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:37.012783 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:37.012672 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:54:53.012661744 +0000 UTC m=+68.741721621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:45.090484 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:45.090459 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n8svj" Jul 13 20:54:49.504024 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.503977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:54:49.507126 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.507107 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 20:54:49.514973 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:49.514956 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 20:54:49.515069 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:49.515038 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:55:53.515016502 +0000 UTC m=+129.244076390 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : secret "metrics-daemon-secret" not found Jul 13 20:54:49.604859 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.604816 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:49.604985 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.604922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:49.607469 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.607449 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 20:54:49.607562 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.607500 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 20:54:49.617308 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.617287 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 20:54:49.617456 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.617438 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03300b5c-abf8-4699-afa0-86ad3a8764c6-original-pull-secret\") pod \"global-pull-secret-syncer-bf8nw\" (UID: \"03300b5c-abf8-4699-afa0-86ad3a8764c6\") " pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:49.627884 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.627865 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zm8n\" (UniqueName: \"kubernetes.io/projected/7dd8b245-47d5-4dd2-b945-a7985d96be9b-kube-api-access-7zm8n\") pod \"network-check-target-qwwxn\" (UID: \"7dd8b245-47d5-4dd2-b945-a7985d96be9b\") " pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:49.743842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.743816 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bf8nw" Jul 13 20:54:49.750813 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.750792 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n7pmh\"" Jul 13 20:54:49.758721 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.758662 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:49.874059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.874030 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bf8nw"] Jul 13 20:54:49.876954 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:54:49.876927 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03300b5c_abf8_4699_afa0_86ad3a8764c6.slice/crio-9f601eae8010505e043bdd3609d5782df682a1b4f9bc5d9da62c0ac99b689fcf WatchSource:0}: Error finding container 9f601eae8010505e043bdd3609d5782df682a1b4f9bc5d9da62c0ac99b689fcf: Status 404 returned error can't find the container with id 9f601eae8010505e043bdd3609d5782df682a1b4f9bc5d9da62c0ac99b689fcf Jul 13 20:54:49.889782 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:49.889744 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qwwxn"] Jul 13 20:54:49.892114 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:54:49.892091 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd8b245_47d5_4dd2_b945_a7985d96be9b.slice/crio-21cd1c2e09e4841d547d121cd52f3bbce9611805e1a5b567f6d32aa1faa6d8e7 WatchSource:0}: Error finding container 21cd1c2e09e4841d547d121cd52f3bbce9611805e1a5b567f6d32aa1faa6d8e7: Status 404 returned error can't find the container with id 21cd1c2e09e4841d547d121cd52f3bbce9611805e1a5b567f6d32aa1faa6d8e7 Jul 13 20:54:50.190268 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:50.190234 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bf8nw" event={"ID":"03300b5c-abf8-4699-afa0-86ad3a8764c6","Type":"ContainerStarted","Data":"9f601eae8010505e043bdd3609d5782df682a1b4f9bc5d9da62c0ac99b689fcf"} Jul 13 20:54:50.191253 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:50.191232 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qwwxn" event={"ID":"7dd8b245-47d5-4dd2-b945-a7985d96be9b","Type":"ContainerStarted","Data":"21cd1c2e09e4841d547d121cd52f3bbce9611805e1a5b567f6d32aa1faa6d8e7"} Jul 13 20:54:53.033379 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.033289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:54:53.033379 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.033346 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033434 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033444 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.033479 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033490 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:55:25.033475779 +0000 UTC m=+100.762535660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033441 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033566 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:55:25.033551817 +0000 UTC m=+100.762611705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033586 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:54:53.033823 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:54:53.033638 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:55:25.033626928 +0000 UTC m=+100.762686806 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:54:53.201902 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.201862 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qwwxn" event={"ID":"7dd8b245-47d5-4dd2-b945-a7985d96be9b","Type":"ContainerStarted","Data":"ac5809db5b0501d1d54071975f67f664844ad91924616542dc26533d7ab6fe02"} Jul 13 20:54:53.202091 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.202018 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:54:53.219105 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:53.218958 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qwwxn" podStartSLOduration=65.364779504 podStartE2EDuration="1m8.218941858s" totalImagesPullingTime="2.854162354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:53.218448267 +0000 UTC m=+68.947508170" watchObservedRunningTime="2026-07-13 20:54:53.218941858 +0000 UTC m=+68.948001759" Jul 13 20:54:55.209358 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:55.209315 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bf8nw" event={"ID":"03300b5c-abf8-4699-afa0-86ad3a8764c6","Type":"ContainerStarted","Data":"25b18931b8b59b04632a0b570789c649e9a86ec8dc42d544c2646628e44c7637"} Jul 13 20:54:55.224636 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:54:55.224584 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bf8nw" podStartSLOduration=64.639478186 podStartE2EDuration="1m9.224564928s" totalImagesPullingTime="4.585086742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:55.223177368 +0000 UTC m=+70.952237267" watchObservedRunningTime="2026-07-13 20:54:55.224564928 +0000 UTC m=+70.953624829" Jul 13 20:55:24.207188 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:55:24.207107 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qwwxn" Jul 13 20:55:25.066256 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:55:25.066215 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:55:25.066256 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:55:25.066258 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:55:25.066299 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066356 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066387 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066421 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:56:29.066404898 +0000 UTC m=+164.795464776 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066435 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:56:29.06642893 +0000 UTC m=+164.795488808 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066447 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066454 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:55:25.066532 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:25.066478 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:56:29.06646762 +0000 UTC m=+164.795527501 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:55:53.569365 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:55:53.569315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:55:53.569854 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:53.569458 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 20:55:53.569854 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:55:53.569529 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:57:55.569512323 +0000 UTC m=+251.298572200 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : secret "metrics-daemon-secret" not found Jul 13 20:56:24.153738 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:24.153638 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" podUID="dadf0e19-20a3-4890-bab4-7a2a9b1c147d" Jul 13 20:56:24.168024 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:24.168000 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" podUID="ae581f63-2c88-48ea-acda-101e930ba694" Jul 13 20:56:24.184152 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:24.184127 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-252h7" podUID="9ce9d425-a05e-4431-9c38-d928e5728715" Jul 13 20:56:24.454678 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:24.454597 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-252h7" Jul 13 20:56:24.454849 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:24.454597 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:56:24.454849 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:24.454603 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:56:24.838942 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:24.838857 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:56:29.123862 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:29.123811 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:29.123901 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.123939 2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:29.123950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124009 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert podName:dadf0e19-20a3-4890-bab4-7a2a9b1c147d nodeName:}" failed. No retries permitted until 2026-07-13 20:58:31.123988772 +0000 UTC m=+286.853048669 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-hx2s6" (UID: "dadf0e19-20a3-4890-bab4-7a2a9b1c147d") : secret "networking-console-plugin-cert" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124035 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124047 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c86cb8dcd-fbt8m: secret "image-registry-tls" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124046 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124093 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls podName:ae581f63-2c88-48ea-acda-101e930ba694 nodeName:}" failed. No retries permitted until 2026-07-13 20:58:31.124078983 +0000 UTC m=+286.853138876 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls") pod "image-registry-c86cb8dcd-fbt8m" (UID: "ae581f63-2c88-48ea-acda-101e930ba694") : secret "image-registry-tls" not found Jul 13 20:56:29.124248 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:56:29.124110 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls podName:9ce9d425-a05e-4431-9c38-d928e5728715 nodeName:}" failed. No retries permitted until 2026-07-13 20:58:31.124101463 +0000 UTC m=+286.853161341 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls") pod "dns-default-252h7" (UID: "9ce9d425-a05e-4431-9c38-d928e5728715") : secret "dns-default-metrics-tls" not found Jul 13 20:56:30.473078 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.473050 2561 generic.go:356] "Generic (PLEG): container finished" podID="085539d5-135c-42ef-81a1-98b93d9ea574" containerID="df1a4185d2b421f23769b56618bee07521fb2f8ec1fd56e6eeb4497deb3d6d28" exitCode=1 Jul 13 20:56:30.473429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.473139 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerDied","Data":"df1a4185d2b421f23769b56618bee07521fb2f8ec1fd56e6eeb4497deb3d6d28"} Jul 13 20:56:30.473545 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.473529 2561 scope.go:117] "RemoveContainer" containerID="df1a4185d2b421f23769b56618bee07521fb2f8ec1fd56e6eeb4497deb3d6d28" Jul 13 20:56:30.474614 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.474594 2561 generic.go:356] "Generic (PLEG): container finished" podID="3530f93f-59b6-47aa-8bac-5831e3ec5a04" containerID="5b163b0c29f0964881ee1a486c50bbec7853e4690d0a66c37bc07e3d72e20a1f" exitCode=255 Jul 13 20:56:30.474720 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.474643 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerDied","Data":"5b163b0c29f0964881ee1a486c50bbec7853e4690d0a66c37bc07e3d72e20a1f"} Jul 13 20:56:30.474948 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:30.474932 2561 scope.go:117] "RemoveContainer" containerID="5b163b0c29f0964881ee1a486c50bbec7853e4690d0a66c37bc07e3d72e20a1f" Jul 13 20:56:31.140181 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:31.140149 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:56:31.479304 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:31.479224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerStarted","Data":"b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e"} Jul 13 20:56:31.479718 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:31.479395 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:56:31.480105 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:31.480088 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:56:31.481000 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:31.480981 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerStarted","Data":"82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f"} Jul 13 20:56:36.828323 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:56:36.828276 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:57:55.628302 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:57:55.628265 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:57:55.628806 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:57:55.628396 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 13 20:57:55.628806 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:57:55.628459 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs podName:2c838c29-7ff1-4577-ad7e-209822026fb9 nodeName:}" failed. No retries permitted until 2026-07-13 20:59:57.628438197 +0000 UTC m=+373.357498080 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs") pod "network-metrics-daemon-97vfm" (UID: "2c838c29-7ff1-4577-ad7e-209822026fb9") : secret "metrics-daemon-secret" not found Jul 13 20:58:08.768404 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.768365 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9"] Jul 13 20:58:08.770424 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.770400 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" Jul 13 20:58:08.773038 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.773020 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 13 20:58:08.773156 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.773033 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 13 20:58:08.774016 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.773997 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kwmtb\"" Jul 13 20:58:08.782324 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.782304 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9"] Jul 13 20:58:08.820113 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.820082 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wnk\" (UniqueName: \"kubernetes.io/projected/a6257856-68e0-4dd0-bda7-161972d9bc9f-kube-api-access-m5wnk\") pod \"migrator-7f59b54f6c-dn9r9\" (UID: \"a6257856-68e0-4dd0-bda7-161972d9bc9f\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" Jul 13 20:58:08.920984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.920939 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wnk\" (UniqueName: \"kubernetes.io/projected/a6257856-68e0-4dd0-bda7-161972d9bc9f-kube-api-access-m5wnk\") pod \"migrator-7f59b54f6c-dn9r9\" (UID: \"a6257856-68e0-4dd0-bda7-161972d9bc9f\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" Jul 13 20:58:08.928556 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:08.928526 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wnk\" (UniqueName: \"kubernetes.io/projected/a6257856-68e0-4dd0-bda7-161972d9bc9f-kube-api-access-m5wnk\") pod \"migrator-7f59b54f6c-dn9r9\" (UID: \"a6257856-68e0-4dd0-bda7-161972d9bc9f\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" Jul 13 20:58:09.080004 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.079919 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" Jul 13 20:58:09.125147 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.124444 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-27wmt"] Jul 13 20:58:09.127358 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.127337 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.129920 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.129893 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 20:58:09.130156 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.130140 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 20:58:09.130741 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.130723 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 20:58:09.130741 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.130722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pgh4v\"" Jul 13 20:58:09.130741 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.130771 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 20:58:09.142908 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.142835 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-27wmt"] Jul 13 20:58:09.198988 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.198883 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9"] Jul 13 20:58:09.201385 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:09.201356 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6257856_68e0_4dd0_bda7_161972d9bc9f.slice/crio-35d6167d95a7ea0704814f7515e60822cf53924de7b11fefbe39aa4531fca05f WatchSource:0}: Error finding container 35d6167d95a7ea0704814f7515e60822cf53924de7b11fefbe39aa4531fca05f: Status 404 returned error can't find the container with id 35d6167d95a7ea0704814f7515e60822cf53924de7b11fefbe39aa4531fca05f Jul 13 20:58:09.223330 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.223309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-data-volume\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.223421 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.223335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.223421 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.223389 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgjm\" (UniqueName: \"kubernetes.io/projected/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-api-access-xvgjm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.223505 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.223431 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-crio-socket\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.223505 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.223449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324369 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324335 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgjm\" (UniqueName: \"kubernetes.io/projected/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-api-access-xvgjm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324408 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-crio-socket\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324470 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-data-volume\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324493 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324693 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324537 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-crio-socket\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.324693 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.324582 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 20:58:09.324693 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.324599 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 20:58:09.324693 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.324649 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:09.82463034 +0000 UTC m=+265.553690221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : secret "insights-runtime-extractor-tls" not found Jul 13 20:58:09.324693 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.324670 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:09.82466018 +0000 UTC m=+265.553720058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : configmap "kube-rbac-proxy" not found Jul 13 20:58:09.324898 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.324881 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-data-volume\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.334612 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.334560 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgjm\" (UniqueName: \"kubernetes.io/projected/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-api-access-xvgjm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.752780 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.752731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" event={"ID":"a6257856-68e0-4dd0-bda7-161972d9bc9f","Type":"ContainerStarted","Data":"35d6167d95a7ea0704814f7515e60822cf53924de7b11fefbe39aa4531fca05f"} Jul 13 20:58:09.828404 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.828376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.828793 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:09.828413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:09.828793 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.828486 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 20:58:09.828793 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.828514 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 20:58:09.828793 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.828535 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:10.828523 +0000 UTC m=+266.557582877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : configmap "kube-rbac-proxy" not found Jul 13 20:58:09.828793 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:09.828566 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:10.828552282 +0000 UTC m=+266.557612159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : secret "insights-runtime-extractor-tls" not found Jul 13 20:58:10.757479 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:10.757388 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" event={"ID":"a6257856-68e0-4dd0-bda7-161972d9bc9f","Type":"ContainerStarted","Data":"b5c360445cf8e9358ec2a13554285faaa6e7834d2a78fbcc69a6d4623e8459fe"} Jul 13 20:58:10.757479 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:10.757436 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" event={"ID":"a6257856-68e0-4dd0-bda7-161972d9bc9f","Type":"ContainerStarted","Data":"927f4e2bc3d6adc2bf1c9de225f0e7faa42cb21571d2bfca76f1953839532bb0"} Jul 13 20:58:10.773290 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:10.773247 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-dn9r9" podStartSLOduration=1.524815688 podStartE2EDuration="2.773229369s" totalImagesPullingTime="1.248413681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:10.772689438 +0000 UTC m=+266.501749356" watchObservedRunningTime="2026-07-13 20:58:10.773229369 +0000 UTC m=+266.502289269" Jul 13 20:58:10.836954 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:10.836916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:10.837350 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:10.837041 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 20:58:10.837350 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:10.837104 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:12.837084825 +0000 UTC m=+268.566144728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : secret "insights-runtime-extractor-tls" not found Jul 13 20:58:10.837350 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:10.837127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:10.837350 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:10.837233 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 20:58:10.837350 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:10.837281 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:12.837267477 +0000 UTC m=+268.566327367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : configmap "kube-rbac-proxy" not found Jul 13 20:58:12.127957 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:12.127931 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5cd5_bbeddf55-2119-4a5b-94be-b776003f0a3c/dns-node-resolver/0.log" Jul 13 20:58:12.851475 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:12.851434 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:12.851626 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:12.851543 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 20:58:12.851626 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:12.851602 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:16.851588382 +0000 UTC m=+272.580648265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : configmap "kube-rbac-proxy" not found Jul 13 20:58:12.851918 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:12.851642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:12.851918 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:12.851732 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 20:58:12.851918 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:12.851775 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:16.851767026 +0000 UTC m=+272.580826903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : secret "insights-runtime-extractor-tls" not found Jul 13 20:58:12.939483 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:12.939454 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8b9q9_e9798223-c3c2-4b30-8311-e9e6780dde4d/node-ca/0.log" Jul 13 20:58:13.729769 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:13.729728 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bgs6t_1833b755-6f8b-4268-b9c9-0471a1755708/serve-healthcheck-canary/0.log" Jul 13 20:58:14.128719 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:14.128693 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dn9r9_a6257856-68e0-4dd0-bda7-161972d9bc9f/migrator/0.log" Jul 13 20:58:14.328029 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:14.327999 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dn9r9_a6257856-68e0-4dd0-bda7-161972d9bc9f/graceful-termination/0.log" Jul 13 20:58:16.883339 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:16.883283 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:16.883776 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:16.883352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:16.883776 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:16.883431 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 20:58:16.883776 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:16.883461 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 20:58:16.883776 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:16.883491 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:24.883477083 +0000 UTC m=+280.612536961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : secret "insights-runtime-extractor-tls" not found Jul 13 20:58:16.883776 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:16.883514 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm podName:17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a nodeName:}" failed. No retries permitted until 2026-07-13 20:58:24.883498694 +0000 UTC m=+280.612558574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-27wmt" (UID: "17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a") : configmap "kube-rbac-proxy" not found Jul 13 20:58:24.951001 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:24.950965 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:24.951001 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:24.951006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:24.951568 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:24.951546 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:24.953190 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:24.953169 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-27wmt\" (UID: \"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a\") " pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:25.039382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:25.039335 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-27wmt" Jul 13 20:58:25.158019 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:25.157996 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-27wmt"] Jul 13 20:58:25.160287 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:25.160254 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f9265c_0aec_4eb5_9e2a_df92bcf1ef1a.slice/crio-94eb09fa8bd50bb718d11699f20908a0b4575d5064c0076f52b933a228ae111a WatchSource:0}: Error finding container 94eb09fa8bd50bb718d11699f20908a0b4575d5064c0076f52b933a228ae111a: Status 404 returned error can't find the container with id 94eb09fa8bd50bb718d11699f20908a0b4575d5064c0076f52b933a228ae111a Jul 13 20:58:25.801821 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:25.801784 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27wmt" event={"ID":"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a","Type":"ContainerStarted","Data":"94eb09fa8bd50bb718d11699f20908a0b4575d5064c0076f52b933a228ae111a"} Jul 13 20:58:27.455705 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:27.455658 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" podUID="dadf0e19-20a3-4890-bab4-7a2a9b1c147d" Jul 13 20:58:27.456213 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:27.455658 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-252h7" podUID="9ce9d425-a05e-4431-9c38-d928e5728715" Jul 13 20:58:27.456213 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:27.455671 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" podUID="ae581f63-2c88-48ea-acda-101e930ba694" Jul 13 20:58:27.807984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:27.807907 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:58:27.807984 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:27.807936 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:27.808197 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:27.808096 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-252h7" Jul 13 20:58:29.815214 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:29.815189 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27wmt" event={"ID":"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a","Type":"ContainerStarted","Data":"754eac57ab0f12727dcd3654842f3e2e9be1c76ae6c8542f5027a3b2b3084680"} Jul 13 20:58:30.820902 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.820871 2561 generic.go:356] "Generic (PLEG): container finished" podID="085539d5-135c-42ef-81a1-98b93d9ea574" containerID="b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e" exitCode=1 Jul 13 20:58:30.821350 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.820956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerDied","Data":"b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e"} Jul 13 20:58:30.821350 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.821036 2561 scope.go:117] "RemoveContainer" containerID="df1a4185d2b421f23769b56618bee07521fb2f8ec1fd56e6eeb4497deb3d6d28" Jul 13 20:58:30.821465 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.821384 2561 scope.go:117] "RemoveContainer" containerID="b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e" Jul 13 20:58:30.821767 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:30.821719 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5668c786cf-l9kdg_open-cluster-management-agent-addon(085539d5-135c-42ef-81a1-98b93d9ea574)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" podUID="085539d5-135c-42ef-81a1-98b93d9ea574" Jul 13 20:58:30.823255 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.823224 2561 generic.go:356] "Generic (PLEG): container finished" podID="3530f93f-59b6-47aa-8bac-5831e3ec5a04" containerID="82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f" exitCode=255 Jul 13 20:58:30.823382 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.823296 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerDied","Data":"82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f"} Jul 13 20:58:30.825552 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.825527 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27wmt" event={"ID":"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a","Type":"ContainerStarted","Data":"b9d21ce89d1d1b4708f9ceac7809e0dd5fe37a845d807ce1af16a79a2c1e3286"} Jul 13 20:58:30.831087 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.831066 2561 scope.go:117] "RemoveContainer" containerID="82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f" Jul 13 20:58:30.831250 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:30.831228 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k_open-cluster-management-agent-addon(3530f93f-59b6-47aa-8bac-5831e3ec5a04)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" podUID="3530f93f-59b6-47aa-8bac-5831e3ec5a04" Jul 13 20:58:30.832564 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:30.832546 2561 scope.go:117] "RemoveContainer" containerID="5b163b0c29f0964881ee1a486c50bbec7853e4690d0a66c37bc07e3d72e20a1f" Jul 13 20:58:31.201688 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.201654 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:58:31.201895 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.201702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:58:31.201895 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.201793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:31.204711 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.204684 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ce9d425-a05e-4431-9c38-d928e5728715-metrics-tls\") pod \"dns-default-252h7\" (UID: \"9ce9d425-a05e-4431-9c38-d928e5728715\") " pod="openshift-dns/dns-default-252h7" Jul 13 20:58:31.204856 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.204792 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"image-registry-c86cb8dcd-fbt8m\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:31.204856 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.204799 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dadf0e19-20a3-4890-bab4-7a2a9b1c147d-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-hx2s6\" (UID: \"dadf0e19-20a3-4890-bab4-7a2a9b1c147d\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:58:31.412763 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.412710 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f67kb\"" Jul 13 20:58:31.412763 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.412711 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4gffg\"" Jul 13 20:58:31.412999 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.412770 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bwbzd\"" Jul 13 20:58:31.420249 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.420229 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:31.420372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.420306 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" Jul 13 20:58:31.420372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.420319 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-252h7" Jul 13 20:58:31.479778 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.479697 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:58:31.831958 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:31.831880 2561 scope.go:117] "RemoveContainer" containerID="b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e" Jul 13 20:58:31.832408 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:31.832072 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5668c786cf-l9kdg_open-cluster-management-agent-addon(085539d5-135c-42ef-81a1-98b93d9ea574)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" podUID="085539d5-135c-42ef-81a1-98b93d9ea574" Jul 13 20:58:32.403028 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.402996 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-hx2s6"] Jul 13 20:58:32.406216 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:32.406182 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadf0e19_20a3_4890_bab4_7a2a9b1c147d.slice/crio-ae6741de08c790df4b975f9fc8f086a94f270a43a9ad8dcbb2205d46ead9dba3 WatchSource:0}: Error finding container ae6741de08c790df4b975f9fc8f086a94f270a43a9ad8dcbb2205d46ead9dba3: Status 404 returned error can't find the container with id ae6741de08c790df4b975f9fc8f086a94f270a43a9ad8dcbb2205d46ead9dba3 Jul 13 20:58:32.626429 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.626406 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-252h7"] Jul 13 20:58:32.628899 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:32.628871 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce9d425_a05e_4431_9c38_d928e5728715.slice/crio-5b4591bcbce7e1697708ac7cfde39eb0f9b0db07a9e81443e631137baaae9247 WatchSource:0}: Error finding container 5b4591bcbce7e1697708ac7cfde39eb0f9b0db07a9e81443e631137baaae9247: Status 404 returned error can't find the container with id 5b4591bcbce7e1697708ac7cfde39eb0f9b0db07a9e81443e631137baaae9247 Jul 13 20:58:32.629524 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.629506 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:58:32.631671 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:32.631619 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae581f63_2c88_48ea_acda_101e930ba694.slice/crio-26fd3fe8b5ad47c2c17899c4a9f1ef9045431633b698b807668d7b60bd8c51ac WatchSource:0}: Error finding container 26fd3fe8b5ad47c2c17899c4a9f1ef9045431633b698b807668d7b60bd8c51ac: Status 404 returned error can't find the container with id 26fd3fe8b5ad47c2c17899c4a9f1ef9045431633b698b807668d7b60bd8c51ac Jul 13 20:58:32.838986 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.838939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-27wmt" event={"ID":"17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a","Type":"ContainerStarted","Data":"bfce6d3c32c8f9b1f70c48c1782563773991df4b1a23dc44685d0fdccdbdb88b"} Jul 13 20:58:32.840247 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.840225 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" event={"ID":"dadf0e19-20a3-4890-bab4-7a2a9b1c147d","Type":"ContainerStarted","Data":"ae6741de08c790df4b975f9fc8f086a94f270a43a9ad8dcbb2205d46ead9dba3"} Jul 13 20:58:32.841359 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.841333 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-252h7" event={"ID":"9ce9d425-a05e-4431-9c38-d928e5728715","Type":"ContainerStarted","Data":"5b4591bcbce7e1697708ac7cfde39eb0f9b0db07a9e81443e631137baaae9247"} Jul 13 20:58:32.842761 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.842735 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" event={"ID":"ae581f63-2c88-48ea-acda-101e930ba694","Type":"ContainerStarted","Data":"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d"} Jul 13 20:58:32.842869 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.842784 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" event={"ID":"ae581f63-2c88-48ea-acda-101e930ba694","Type":"ContainerStarted","Data":"26fd3fe8b5ad47c2c17899c4a9f1ef9045431633b698b807668d7b60bd8c51ac"} Jul 13 20:58:32.842948 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.842902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:32.856074 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.856022 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-27wmt" podStartSLOduration=16.823643697 podStartE2EDuration="23.856004359s" totalImagesPullingTime="7.032360662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:58:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:32.855774492 +0000 UTC m=+288.584834387" watchObservedRunningTime="2026-07-13 20:58:32.856004359 +0000 UTC m=+288.585064260" Jul 13 20:58:32.873949 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:32.873899 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" podStartSLOduration=264.873879846 podStartE2EDuration="4m24.873879846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:32.873079922 +0000 UTC m=+288.602139854" watchObservedRunningTime="2026-07-13 20:58:32.873879846 +0000 UTC m=+288.602939748" Jul 13 20:58:34.850444 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.850410 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-252h7" event={"ID":"9ce9d425-a05e-4431-9c38-d928e5728715","Type":"ContainerStarted","Data":"10bd73b728f301b37109fc2b7a59f4005c07bf134a7b36e9c3e8c71846336fc3"} Jul 13 20:58:34.850851 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.850460 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-252h7" event={"ID":"9ce9d425-a05e-4431-9c38-d928e5728715","Type":"ContainerStarted","Data":"0367e27a9040b6c63f8534573e518036ce4cca33b61a63248a7fc075d60357a8"} Jul 13 20:58:34.850851 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.850511 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-252h7" Jul 13 20:58:34.851922 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.851901 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" event={"ID":"dadf0e19-20a3-4890-bab4-7a2a9b1c147d","Type":"ContainerStarted","Data":"619109b52a525725c1e5e9fa737af2a341c23adc066c7e808d89871d32d5ba73"} Jul 13 20:58:34.868430 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.868388 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-252h7" podStartSLOduration=251.913126599 podStartE2EDuration="4m13.868374852s" totalImagesPullingTime="1.955248253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:34.866714936 +0000 UTC m=+290.595774836" watchObservedRunningTime="2026-07-13 20:58:34.868374852 +0000 UTC m=+290.597434752" Jul 13 20:58:34.883859 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:34.883818 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-hx2s6" podStartSLOduration=279.496801556 podStartE2EDuration="4m40.883803638s" totalImagesPullingTime="1.387002082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:34.883768997 +0000 UTC m=+290.612828902" watchObservedRunningTime="2026-07-13 20:58:34.883803638 +0000 UTC m=+290.612863540" Jul 13 20:58:35.499791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:35.499744 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" Jul 13 20:58:35.500059 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:35.500047 2561 scope.go:117] "RemoveContainer" containerID="82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f" Jul 13 20:58:35.500198 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:35.500180 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k_open-cluster-management-agent-addon(3530f93f-59b6-47aa-8bac-5831e3ec5a04)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" podUID="3530f93f-59b6-47aa-8bac-5831e3ec5a04" Jul 13 20:58:35.508313 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:35.508296 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:58:35.508581 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:35.508570 2561 scope.go:117] "RemoveContainer" containerID="b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e" Jul 13 20:58:35.508705 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:35.508687 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-5668c786cf-l9kdg_open-cluster-management-agent-addon(085539d5-135c-42ef-81a1-98b93d9ea574)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" podUID="085539d5-135c-42ef-81a1-98b93d9ea574" Jul 13 20:58:36.446176 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.446136 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq"] Jul 13 20:58:36.449249 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.449226 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:36.452456 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.452438 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 20:58:36.452534 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.452438 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6mtnb\"" Jul 13 20:58:36.461983 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.461960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq"] Jul 13 20:58:36.543830 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.543800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-hmknq\" (UID: \"8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:36.546884 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.546859 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-nvct7"] Jul 13 20:58:36.549041 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.549026 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:36.552054 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.552037 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vffbl\"" Jul 13 20:58:36.552282 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.552261 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 20:58:36.552467 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.552450 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 20:58:36.565103 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.565082 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nvct7"] Jul 13 20:58:36.644992 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.644963 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-hmknq\" (UID: \"8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:36.645121 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.645009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc4cn\" (UniqueName: \"kubernetes.io/projected/a27239ed-2691-42d5-b547-3ef63e441ce9-kube-api-access-qc4cn\") pod \"downloads-68c9cbb497-nvct7\" (UID: \"a27239ed-2691-42d5-b547-3ef63e441ce9\") " pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:36.647298 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.647277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-hmknq\" (UID: \"8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:36.745402 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.745327 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc4cn\" (UniqueName: \"kubernetes.io/projected/a27239ed-2691-42d5-b547-3ef63e441ce9-kube-api-access-qc4cn\") pod \"downloads-68c9cbb497-nvct7\" (UID: \"a27239ed-2691-42d5-b547-3ef63e441ce9\") " pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:36.753831 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.753801 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc4cn\" (UniqueName: \"kubernetes.io/projected/a27239ed-2691-42d5-b547-3ef63e441ce9-kube-api-access-qc4cn\") pod \"downloads-68c9cbb497-nvct7\" (UID: \"a27239ed-2691-42d5-b547-3ef63e441ce9\") " pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:36.772786 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.772745 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:36.857152 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.857131 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:36.898641 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.898577 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq"] Jul 13 20:58:36.902222 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:36.902189 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc7bc47_368b_4ad9_a2f4_1f10f6e427cf.slice/crio-2c626437c113d2514a67115c43b5a1fd5c141d9aa912096cb00a04c5ce800e72 WatchSource:0}: Error finding container 2c626437c113d2514a67115c43b5a1fd5c141d9aa912096cb00a04c5ce800e72: Status 404 returned error can't find the container with id 2c626437c113d2514a67115c43b5a1fd5c141d9aa912096cb00a04c5ce800e72 Jul 13 20:58:36.980056 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:36.980011 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nvct7"] Jul 13 20:58:36.982278 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:36.982251 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27239ed_2691_42d5_b547_3ef63e441ce9.slice/crio-acd35a7f4ca717fabab1d70d964dd099130ecfe10f6511ede3f4f3e8180475b7 WatchSource:0}: Error finding container acd35a7f4ca717fabab1d70d964dd099130ecfe10f6511ede3f4f3e8180475b7: Status 404 returned error can't find the container with id acd35a7f4ca717fabab1d70d964dd099130ecfe10f6511ede3f4f3e8180475b7 Jul 13 20:58:37.867322 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:37.867249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nvct7" event={"ID":"a27239ed-2691-42d5-b547-3ef63e441ce9","Type":"ContainerStarted","Data":"acd35a7f4ca717fabab1d70d964dd099130ecfe10f6511ede3f4f3e8180475b7"} Jul 13 20:58:37.868526 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:37.868495 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" event={"ID":"8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf","Type":"ContainerStarted","Data":"2c626437c113d2514a67115c43b5a1fd5c141d9aa912096cb00a04c5ce800e72"} Jul 13 20:58:38.874763 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:38.874710 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" event={"ID":"8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf","Type":"ContainerStarted","Data":"1e8718f991b7f9e03fc2e8d5a8e8303a6ca7637b4e82655b821bbec2f58b547e"} Jul 13 20:58:38.875217 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:38.874927 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:38.881139 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:38.881117 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" Jul 13 20:58:38.892434 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:38.892388 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-hmknq" podStartSLOduration=1.821687828 podStartE2EDuration="2.892370206s" totalImagesPullingTime="1.070682378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:38.891295086 +0000 UTC m=+294.620355014" watchObservedRunningTime="2026-07-13 20:58:38.892370206 +0000 UTC m=+294.621430106" Jul 13 20:58:39.830023 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:39.829975 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-97vfm" podUID="2c838c29-7ff1-4577-ad7e-209822026fb9" Jul 13 20:58:44.696504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:44.696477 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:58:44.700089 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:44.700068 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 20:58:44.700367 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:44.700348 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 20:58:44.858506 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:44.858477 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-252h7" Jul 13 20:58:45.734673 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.734637 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:58:45.737159 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.737138 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.740701 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740667 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 20:58:45.740842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740702 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 20:58:45.740842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740702 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 20:58:45.740842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740790 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8vlv\"" Jul 13 20:58:45.740842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740702 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 20:58:45.740842 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.740834 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 20:58:45.746945 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.746910 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 20:58:45.755272 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.755249 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:58:45.819950 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.819919 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820116 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.819980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820116 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.820009 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820116 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.820073 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820116 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.820099 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820285 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.820171 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.820285 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.820192 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hpbs\" (UniqueName: \"kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921248 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921553 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921258 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hpbs\" (UniqueName: \"kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921645 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921602 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921714 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921714 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921871 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921746 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.921871 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.921803 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.922184 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.922155 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.922482 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.922455 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.922641 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.922615 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.922802 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.922779 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.924713 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.924693 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.924903 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.924884 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:45.936108 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.936088 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-62chb"] Jul 13 20:58:45.939075 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.939057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:45.945783 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.945765 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 20:58:45.945890 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.945776 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gxcf2\"" Jul 13 20:58:45.945890 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.945745 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 20:58:45.946108 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.946087 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 20:58:45.946224 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.946172 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 20:58:45.946347 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.946323 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 20:58:45.946765 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.946711 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 20:58:45.949624 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:45.949606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hpbs\" (UniqueName: \"kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs\") pod \"console-846cc9c875-jpgcm\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:46.022374 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022299 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbggc\" (UniqueName: \"kubernetes.io/projected/7e464e29-fb88-4184-a656-6ca19eb4d4ab-kube-api-access-qbggc\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022374 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-textfile\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022401 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-wtmp\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022433 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-metrics-client-ca\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022506 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-root\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022538 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022527 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-sys\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022794 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022545 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.022794 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.022572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-tls\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.048637 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.048613 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:58:46.123505 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123439 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbggc\" (UniqueName: \"kubernetes.io/projected/7e464e29-fb88-4184-a656-6ca19eb4d4ab-kube-api-access-qbggc\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123505 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123490 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-textfile\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123530 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-wtmp\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123596 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-metrics-client-ca\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123641 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-root\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123675 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-sys\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.123725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-sys\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123788 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-root\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123871 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-tls\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.123982 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-wtmp\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124254 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.124194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-textfile\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124513 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.124467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.124726 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.124520 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e464e29-fb88-4184-a656-6ca19eb4d4ab-metrics-client-ca\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.126546 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.126507 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-tls\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.127396 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.127356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e464e29-fb88-4184-a656-6ca19eb4d4ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.131295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.131275 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbggc\" (UniqueName: \"kubernetes.io/projected/7e464e29-fb88-4184-a656-6ca19eb4d4ab-kube-api-access-qbggc\") pod \"node-exporter-62chb\" (UID: \"7e464e29-fb88-4184-a656-6ca19eb4d4ab\") " pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.190807 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.190782 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:58:46.193037 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:46.193009 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ab791a_f87d_444c_9735_5012f06316f4.slice/crio-03d58d0e82d29b3e0bce6bdd6441db3d94457e0950f39cf7aefa18b8f093c0f3 WatchSource:0}: Error finding container 03d58d0e82d29b3e0bce6bdd6441db3d94457e0950f39cf7aefa18b8f093c0f3: Status 404 returned error can't find the container with id 03d58d0e82d29b3e0bce6bdd6441db3d94457e0950f39cf7aefa18b8f093c0f3 Jul 13 20:58:46.195293 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.195276 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 20:58:46.250110 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.250084 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-62chb" Jul 13 20:58:46.261342 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:58:46.261313 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e464e29_fb88_4184_a656_6ca19eb4d4ab.slice/crio-ceac30d2cf7d0595ae1f26c6a4bb068a78dd86c47e271f4c58619a0ec2c18141 WatchSource:0}: Error finding container ceac30d2cf7d0595ae1f26c6a4bb068a78dd86c47e271f4c58619a0ec2c18141: Status 404 returned error can't find the container with id ceac30d2cf7d0595ae1f26c6a4bb068a78dd86c47e271f4c58619a0ec2c18141 Jul 13 20:58:46.828514 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.828476 2561 scope.go:117] "RemoveContainer" containerID="b55813bf12d8329a8acfe55ef8ad2d164ee096b3758394ad1aa3dc1eeb78f49e" Jul 13 20:58:46.906423 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.906380 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62chb" event={"ID":"7e464e29-fb88-4184-a656-6ca19eb4d4ab","Type":"ContainerStarted","Data":"ceac30d2cf7d0595ae1f26c6a4bb068a78dd86c47e271f4c58619a0ec2c18141"} Jul 13 20:58:46.907578 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:46.907555 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846cc9c875-jpgcm" event={"ID":"58ab791a-f87d-444c-9735-5012f06316f4","Type":"ContainerStarted","Data":"03d58d0e82d29b3e0bce6bdd6441db3d94457e0950f39cf7aefa18b8f093c0f3"} Jul 13 20:58:48.837526 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:48.837491 2561 scope.go:117] "RemoveContainer" containerID="82ca2496820a2ff72ba7eb2f29a0e753669f3a04f4e55f33410bd053a165530f" Jul 13 20:58:50.681360 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.681330 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj"] Jul 13 20:58:50.684115 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.684093 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:50.687224 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.687199 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cjb69\"" Jul 13 20:58:50.687330 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.687224 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 13 20:58:50.687737 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.687717 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:58:50.694114 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.694094 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj"] Jul 13 20:58:50.765565 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.765535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfrz\" (UniqueName: \"kubernetes.io/projected/38d4da58-88f3-4733-992a-556b1e6ebc34-kube-api-access-vxfrz\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:50.765731 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.765634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:50.866933 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.866897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:50.867120 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.866966 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfrz\" (UniqueName: \"kubernetes.io/projected/38d4da58-88f3-4733-992a-556b1e6ebc34-kube-api-access-vxfrz\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:50.867120 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:50.867054 2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 13 20:58:50.867120 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:58:50.867118 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert podName:38d4da58-88f3-4733-992a-556b1e6ebc34 nodeName:}" failed. No retries permitted until 2026-07-13 20:58:51.367100824 +0000 UTC m=+307.096160708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-5c9lj" (UID: "38d4da58-88f3-4733-992a-556b1e6ebc34") : secret "monitoring-plugin-cert" not found Jul 13 20:58:50.878372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:50.878340 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfrz\" (UniqueName: \"kubernetes.io/projected/38d4da58-88f3-4733-992a-556b1e6ebc34-kube-api-access-vxfrz\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:51.371092 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:51.371052 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:51.373991 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:51.373964 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38d4da58-88f3-4733-992a-556b1e6ebc34-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5c9lj\" (UID: \"38d4da58-88f3-4733-992a-556b1e6ebc34\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:51.425642 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:51.425612 2561 patch_prober.go:28] interesting pod/image-registry-c86cb8dcd-fbt8m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 13 20:58:51.425788 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:51.425665 2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" podUID="ae581f63-2c88-48ea-acda-101e930ba694" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 20:58:51.604888 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:51.604840 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:52.135521 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.135487 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 20:58:52.140068 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.140046 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.142425 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.142404 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 20:58:52.142547 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.142490 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 20:58:52.142664 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.142645 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 20:58:52.142736 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.142689 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 20:58:52.142736 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.142706 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 20:58:52.143339 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.143314 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bppug4s7a7q4k\"" Jul 13 20:58:52.143445 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.143401 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 20:58:52.143445 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.143434 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 20:58:52.143947 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.143918 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 20:58:52.144062 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.144011 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 20:58:52.144062 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.144017 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 20:58:52.144171 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.144107 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 20:58:52.144334 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.144314 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xkk8s\"" Jul 13 20:58:52.145355 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.145214 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 20:58:52.145355 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.145248 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 20:58:52.145355 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.145248 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 20:58:52.147062 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.147030 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 20:58:52.157725 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.157701 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 20:58:52.177222 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177201 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177319 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177237 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177319 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177272 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177319 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177473 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177346 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177473 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177473 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177473 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177443 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177661 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177661 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177534 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177661 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177562 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177661 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177661 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177854 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177854 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177763 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177854 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177800 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.177854 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.178009 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.178009 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177926 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.178009 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.177988 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.278847 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278803 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.278847 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278843 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279109 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278874 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279109 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278904 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279109 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278922 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279109 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278952 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279109 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.278997 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279368 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279262 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279368 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279304 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279368 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279339 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279389 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279448 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279481 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279515 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279506 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279670 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279708 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.279791 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.279737 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.280226 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.280196 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.281607 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.280479 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.281607 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.281286 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.282308 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.282247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.282645 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.282623 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.283654 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.283624 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.283850 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.283826 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.284703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.283986 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.284703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.284418 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.284703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.284603 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.284703 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.284669 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.285410 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.285356 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.285511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.285420 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.285851 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.285804 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.285851 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.285805 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.286003 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.285959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.286504 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.286469 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.290457 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.290435 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d\") pod \"prometheus-k8s-0\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:52.452100 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:52.452013 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:58:53.463490 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.463358 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 20:58:53.479241 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.479100 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj"] Jul 13 20:58:53.829008 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.828854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:58:53.851904 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.851729 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:58:53.939685 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.939308 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" event={"ID":"38d4da58-88f3-4733-992a-556b1e6ebc34","Type":"ContainerStarted","Data":"8d0945649bc15ddbdad2d49f6862535903dc16d347b57ec85c8ff19d5cc89429"} Jul 13 20:58:53.943264 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.943219 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" event={"ID":"085539d5-135c-42ef-81a1-98b93d9ea574","Type":"ContainerStarted","Data":"4f33e4af609c41e53c180a551a2a6a3940b88a79e7b5a028d092c5e81c6ff787"} Jul 13 20:58:53.943914 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.943888 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:58:53.947022 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.945780 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5668c786cf-l9kdg" Jul 13 20:58:53.948595 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.948082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7d9b6b64-l6j2k" event={"ID":"3530f93f-59b6-47aa-8bac-5831e3ec5a04","Type":"ContainerStarted","Data":"44ed734ec257fee1aa630f803bfca9e95d0a090f4551483152d9e1f544bbee58"} Jul 13 20:58:53.951298 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.951241 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nvct7" event={"ID":"a27239ed-2691-42d5-b547-3ef63e441ce9","Type":"ContainerStarted","Data":"6c0bb45096e0731d637746d00aa4612625e916f74745ae95a2132996a6f230f8"} Jul 13 20:58:53.951575 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.951505 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:53.953613 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.953586 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"2e7a4a5a43eccd6e53f35ae3653414c2a150951e9faf1fe112ff6a38d1350a0a"} Jul 13 20:58:53.964421 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.964399 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-nvct7" Jul 13 20:58:53.975459 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:53.974385 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-nvct7" podStartSLOduration=1.578456665 podStartE2EDuration="17.974368063s" totalImagesPullingTime="16.395911398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:53.973575195 +0000 UTC m=+309.702635107" watchObservedRunningTime="2026-07-13 20:58:53.974368063 +0000 UTC m=+309.703427964" Jul 13 20:58:54.963387 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:54.963346 2561 generic.go:356] "Generic (PLEG): container finished" podID="7e464e29-fb88-4184-a656-6ca19eb4d4ab" containerID="3cdd4493ea3ebdc19416e788fc94714c46f62a3c3c7ed8ac7d6d47f94ab49ee3" exitCode=0 Jul 13 20:58:54.963895 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:54.963523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62chb" event={"ID":"7e464e29-fb88-4184-a656-6ca19eb4d4ab","Type":"ContainerDied","Data":"3cdd4493ea3ebdc19416e788fc94714c46f62a3c3c7ed8ac7d6d47f94ab49ee3"} Jul 13 20:58:57.979299 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.979207 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62chb" event={"ID":"7e464e29-fb88-4184-a656-6ca19eb4d4ab","Type":"ContainerStarted","Data":"35637a0e17648444cdd6e5d7110218a29e2f777911e505ff4cee4fbfc21c4668"} Jul 13 20:58:57.979299 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.979287 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62chb" event={"ID":"7e464e29-fb88-4184-a656-6ca19eb4d4ab","Type":"ContainerStarted","Data":"d608df2ad38a55f242edd1b20f1e06ffca19c021140c54ce3db71b3b1b8ceba3"} Jul 13 20:58:57.981055 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.980958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846cc9c875-jpgcm" event={"ID":"58ab791a-f87d-444c-9735-5012f06316f4","Type":"ContainerStarted","Data":"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba"} Jul 13 20:58:57.983361 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.982818 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" exitCode=0 Jul 13 20:58:57.983361 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.982945 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} Jul 13 20:58:57.985229 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.985202 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" event={"ID":"38d4da58-88f3-4733-992a-556b1e6ebc34","Type":"ContainerStarted","Data":"9cd4adbb72af574be877dbb9557d8cfe6217dabbf8e67eb06b8a387cff47a49a"} Jul 13 20:58:57.985457 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.985412 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:57.991479 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.991459 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" Jul 13 20:58:57.996436 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:57.996390 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-62chb" podStartSLOduration=4.980730849 podStartE2EDuration="12.996373701s" totalImagesPullingTime="7.925179631s" totalInitContainerRuntime="90.463221ms" isStatefulPod=true podCreationTimestamp="2026-07-13 20:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:57.996040448 +0000 UTC m=+313.725100367" watchObservedRunningTime="2026-07-13 20:58:57.996373701 +0000 UTC m=+313.725433602" Jul 13 20:58:58.012318 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:58.012274 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846cc9c875-jpgcm" podStartSLOduration=1.5613576949999999 podStartE2EDuration="13.012258924s" totalImagesPullingTime="11.450901229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:58.012187839 +0000 UTC m=+313.741247753" watchObservedRunningTime="2026-07-13 20:58:58.012258924 +0000 UTC m=+313.741318826" Jul 13 20:58:58.027046 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:58.027009 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5c9lj" podStartSLOduration=3.872576562 podStartE2EDuration="8.026995616s" totalImagesPullingTime="4.154419054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:58:58.025789236 +0000 UTC m=+313.754849152" watchObservedRunningTime="2026-07-13 20:58:58.026995616 +0000 UTC m=+313.756055534" Jul 13 20:58:58.536807 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:58:58.536771 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:59:03.008158 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:03.008079 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} Jul 13 20:59:03.008158 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:03.008144 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} Jul 13 20:59:06.023972 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.023938 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} Jul 13 20:59:06.024528 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.024025 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} Jul 13 20:59:06.024528 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.024053 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} Jul 13 20:59:06.024528 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.024077 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerStarted","Data":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} Jul 13 20:59:06.049099 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.049073 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:59:06.052204 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:06.052087 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.006818497 podStartE2EDuration="14.052068472s" totalImagesPullingTime="11.021532155s" totalInitContainerRuntime="23.71782ms" isStatefulPod=false podCreationTimestamp="2026-07-13 20:58:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:59:06.049740555 +0000 UTC m=+321.778800475" watchObservedRunningTime="2026-07-13 20:59:06.052068472 +0000 UTC m=+321.781128372" Jul 13 20:59:07.452510 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:07.452475 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:59:23.007098 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.007060 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-846cc9c875-jpgcm" podUID="58ab791a-f87d-444c-9735-5012f06316f4" containerName="console" containerID="cri-o://a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba" gracePeriod=15 Jul 13 20:59:23.247008 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.246987 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:59:23.357526 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357499 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.357689 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357552 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.357797 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357741 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.357932 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357828 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.357932 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357872 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.357932 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357901 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.357932 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357909 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.358146 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357960 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hpbs\" (UniqueName: \"kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs\") pod \"58ab791a-f87d-444c-9735-5012f06316f4\" (UID: \"58ab791a-f87d-444c-9735-5012f06316f4\") " Jul 13 20:59:23.358146 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.357977 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config" (OuterVolumeSpecName: "console-config") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.358146 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.358117 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.358295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.358195 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.358295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.358242 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-console-config\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.358295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.358262 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-trusted-ca-bundle\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.358295 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.358278 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-oauth-serving-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.360033 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.360012 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:59:23.360119 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.360082 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:59:23.360187 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.360165 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs" (OuterVolumeSpecName: "kube-api-access-6hpbs") pod "58ab791a-f87d-444c-9735-5012f06316f4" (UID: "58ab791a-f87d-444c-9735-5012f06316f4"). InnerVolumeSpecName "kube-api-access-6hpbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:59:23.459405 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.459371 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58ab791a-f87d-444c-9735-5012f06316f4-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.459405 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.459400 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-serving-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.459574 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.459413 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58ab791a-f87d-444c-9735-5012f06316f4-console-oauth-config\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.459574 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.459426 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hpbs\" (UniqueName: \"kubernetes.io/projected/58ab791a-f87d-444c-9735-5012f06316f4-kube-api-access-6hpbs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:23.561122 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.561065 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" podUID="ae581f63-2c88-48ea-acda-101e930ba694" containerName="registry" containerID="cri-o://a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d" gracePeriod=30 Jul 13 20:59:23.799942 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.799918 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:59:23.962867 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.962785 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.962867 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.962835 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963085 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.962883 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963085 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.962926 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963085 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.962955 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwkl7\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963237 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.963087 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963237 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.963129 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963237 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.963185 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted\") pod \"ae581f63-2c88-48ea-acda-101e930ba694\" (UID: \"ae581f63-2c88-48ea-acda-101e930ba694\") " Jul 13 20:59:23.963412 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.963293 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.963537 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.963512 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 20:59:23.965372 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.965341 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:59:23.965555 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.965523 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:59:23.965624 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.965583 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7" (OuterVolumeSpecName: "kube-api-access-rwkl7") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "kube-api-access-rwkl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:59:23.965624 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.965587 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 20:59:23.965695 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.965620 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 20:59:23.971416 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:23.971396 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ae581f63-2c88-48ea-acda-101e930ba694" (UID: "ae581f63-2c88-48ea-acda-101e930ba694"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 20:59:24.064256 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064230 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-image-registry-private-configuration\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064256 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064253 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwkl7\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-kube-api-access-rwkl7\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064262 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-trusted-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064273 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae581f63-2c88-48ea-acda-101e930ba694-installation-pull-secrets\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064281 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae581f63-2c88-48ea-acda-101e930ba694-ca-trust-extracted\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064289 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-registry-tls\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064297 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae581f63-2c88-48ea-acda-101e930ba694-registry-certificates\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.064572 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.064308 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae581f63-2c88-48ea-acda-101e930ba694-bound-sa-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 20:59:24.095223 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.095198 2561 generic.go:356] "Generic (PLEG): container finished" podID="58ab791a-f87d-444c-9735-5012f06316f4" containerID="a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba" exitCode=0 Jul 13 20:59:24.095328 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.095261 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846cc9c875-jpgcm" Jul 13 20:59:24.095328 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.095283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846cc9c875-jpgcm" event={"ID":"58ab791a-f87d-444c-9735-5012f06316f4","Type":"ContainerDied","Data":"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba"} Jul 13 20:59:24.095434 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.095339 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846cc9c875-jpgcm" event={"ID":"58ab791a-f87d-444c-9735-5012f06316f4","Type":"ContainerDied","Data":"03d58d0e82d29b3e0bce6bdd6441db3d94457e0950f39cf7aefa18b8f093c0f3"} Jul 13 20:59:24.095434 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.095371 2561 scope.go:117] "RemoveContainer" containerID="a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba" Jul 13 20:59:24.096684 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.096609 2561 generic.go:356] "Generic (PLEG): container finished" podID="ae581f63-2c88-48ea-acda-101e930ba694" containerID="a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d" exitCode=0 Jul 13 20:59:24.096684 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.096651 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" Jul 13 20:59:24.096684 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.096672 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" event={"ID":"ae581f63-2c88-48ea-acda-101e930ba694","Type":"ContainerDied","Data":"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d"} Jul 13 20:59:24.096825 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.096705 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c86cb8dcd-fbt8m" event={"ID":"ae581f63-2c88-48ea-acda-101e930ba694","Type":"ContainerDied","Data":"26fd3fe8b5ad47c2c17899c4a9f1ef9045431633b698b807668d7b60bd8c51ac"} Jul 13 20:59:24.104688 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.104664 2561 scope.go:117] "RemoveContainer" containerID="a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba" Jul 13 20:59:24.104999 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:59:24.104960 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba\": container with ID starting with a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba not found: ID does not exist" containerID="a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba" Jul 13 20:59:24.105072 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.104993 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba"} err="failed to get container status \"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba\": rpc error: code = NotFound desc = could not find container \"a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba\": container with ID starting with a189bea65a330d317a1a23eaee7c893cc834c9a4b3f3be6fc2e692ce9065faba not found: ID does not exist" Jul 13 20:59:24.105072 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.105030 2561 scope.go:117] "RemoveContainer" containerID="a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d" Jul 13 20:59:24.112817 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.112799 2561 scope.go:117] "RemoveContainer" containerID="a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d" Jul 13 20:59:24.113079 ip-10-0-130-205 kubenswrapper[2561]: E0713 20:59:24.113058 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d\": container with ID starting with a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d not found: ID does not exist" containerID="a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d" Jul 13 20:59:24.113190 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.113082 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d"} err="failed to get container status \"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d\": rpc error: code = NotFound desc = could not find container \"a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d\": container with ID starting with a9cc3ca57f29fe390986b987f56fd4704b17be4f9c42f356e1e2f62b7e25935d not found: ID does not exist" Jul 13 20:59:24.125166 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.125141 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:59:24.132874 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.132855 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c86cb8dcd-fbt8m"] Jul 13 20:59:24.145086 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.145067 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:59:24.152511 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.152490 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846cc9c875-jpgcm"] Jul 13 20:59:24.833472 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.833440 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ab791a-f87d-444c-9735-5012f06316f4" path="/var/lib/kubelet/pods/58ab791a-f87d-444c-9735-5012f06316f4/volumes" Jul 13 20:59:24.833869 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:24.833855 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae581f63-2c88-48ea-acda-101e930ba694" path="/var/lib/kubelet/pods/ae581f63-2c88-48ea-acda-101e930ba694/volumes" Jul 13 20:59:52.452643 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:52.452603 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:59:52.472118 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:52.472094 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:59:53.218192 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:53.218166 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 20:59:57.636816 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:57.636781 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:59:57.639043 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:57.639024 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c838c29-7ff1-4577-ad7e-209822026fb9-metrics-certs\") pod \"network-metrics-daemon-97vfm\" (UID: \"2c838c29-7ff1-4577-ad7e-209822026fb9\") " pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:59:57.732882 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:57.732854 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plnw6\"" Jul 13 20:59:57.740820 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:57.740803 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-97vfm" Jul 13 20:59:57.860735 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:57.860701 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-97vfm"] Jul 13 20:59:57.864246 ip-10-0-130-205 kubenswrapper[2561]: W0713 20:59:57.864221 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c838c29_7ff1_4577_ad7e_209822026fb9.slice/crio-e6b4a894205ecfd28b896de0fb4b783ce776393c2f532e64f6a160cebfc2bf0e WatchSource:0}: Error finding container e6b4a894205ecfd28b896de0fb4b783ce776393c2f532e64f6a160cebfc2bf0e: Status 404 returned error can't find the container with id e6b4a894205ecfd28b896de0fb4b783ce776393c2f532e64f6a160cebfc2bf0e Jul 13 20:59:58.222166 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:58.222136 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97vfm" event={"ID":"2c838c29-7ff1-4577-ad7e-209822026fb9","Type":"ContainerStarted","Data":"e6b4a894205ecfd28b896de0fb4b783ce776393c2f532e64f6a160cebfc2bf0e"} Jul 13 20:59:59.227678 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:59.227597 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97vfm" event={"ID":"2c838c29-7ff1-4577-ad7e-209822026fb9","Type":"ContainerStarted","Data":"55ce8cb1aed2e8c90fbe55822cc8c63b59b52bdc656bbf56c3116e1ddb3221e2"} Jul 13 20:59:59.227678 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:59.227655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-97vfm" event={"ID":"2c838c29-7ff1-4577-ad7e-209822026fb9","Type":"ContainerStarted","Data":"fa3e2d55294bd7b726fc70936d04aaae249d79b596dca2e9bfc84671861a96b9"} Jul 13 20:59:59.244653 ip-10-0-130-205 kubenswrapper[2561]: I0713 20:59:59.244601 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-97vfm" podStartSLOduration=373.244335105 podStartE2EDuration="6m14.244586329s" totalImagesPullingTime="1.000251224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:59:59.24401061 +0000 UTC m=+374.973070522" watchObservedRunningTime="2026-07-13 20:59:59.244586329 +0000 UTC m=+374.973646229" Jul 13 21:00:10.540721 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.540680 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:10.541217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541077 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="prometheus" containerID="cri-o://d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" gracePeriod=600 Jul 13 21:00:10.541217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541091 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy" containerID="cri-o://f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" gracePeriod=600 Jul 13 21:00:10.541217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541133 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-web" containerID="cri-o://3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" gracePeriod=600 Jul 13 21:00:10.541217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541145 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" gracePeriod=600 Jul 13 21:00:10.541396 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541145 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="thanos-sidecar" containerID="cri-o://80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" gracePeriod=600 Jul 13 21:00:10.541396 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.541153 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="config-reloader" containerID="cri-o://9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" gracePeriod=600 Jul 13 21:00:10.778778 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.778739 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:10.940093 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940057 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940093 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940096 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940117 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940134 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940276 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940313 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940339 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940373 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940402 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940435 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940463 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940483 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940495 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940534 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.940596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940549 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940602 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940724 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940783 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940826 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940863 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940871 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940901 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.940939 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1\") pod \"ee4b049b-ae22-45da-9007-b866cd41f4c1\" (UID: \"ee4b049b-ae22-45da-9007-b866cd41f4c1\") " Jul 13 21:00:10.941362 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.941160 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-metrics-client-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:10.941362 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.941178 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:10.941362 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.941194 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:10.943032 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.942966 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.943413 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943392 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.943555 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943396 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config" (OuterVolumeSpecName: "config") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.944502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943468 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d" (OuterVolumeSpecName: "kube-api-access-g669d") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "kube-api-access-g669d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:00:10.944636 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944604 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.944797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943523 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.944913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943521 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out" (OuterVolumeSpecName: "config-out") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:00:10.944913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.943672 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.944913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944083 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.945057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944355 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.945057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944859 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.945057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944887 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:00:10.945057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.944981 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:00:10.945252 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.945236 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:00:10.945619 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.945599 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.946081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.946056 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:10.953777 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:10.953737 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config" (OuterVolumeSpecName: "web-config") pod "ee4b049b-ae22-45da-9007-b866cd41f4c1" (UID: "ee4b049b-ae22-45da-9007-b866cd41f4c1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:00:11.042390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042356 2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-config\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042381 2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-grpc-tls\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042393 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042403 2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-kube-rbac-proxy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042411 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-tls-assets\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042420 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042428 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g669d\" (UniqueName: \"kubernetes.io/projected/ee4b049b-ae22-45da-9007-b866cd41f4c1-kube-api-access-g669d\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042436 2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-metrics-client-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042445 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-config-out\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042453 2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042460 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042469 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-db\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042476 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042484 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042492 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042501 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ee4b049b-ae22-45da-9007-b866cd41f4c1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.042647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.042508 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee4b049b-ae22-45da-9007-b866cd41f4c1-web-config\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273703 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273726 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273732 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273740 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273767 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273779 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" exitCode=0 Jul 13 21:00:11.273797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273785 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273814 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273852 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273906 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273919 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273932 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} Jul 13 21:00:11.274109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.273980 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ee4b049b-ae22-45da-9007-b866cd41f4c1","Type":"ContainerDied","Data":"2e7a4a5a43eccd6e53f35ae3653414c2a150951e9faf1fe112ff6a38d1350a0a"} Jul 13 21:00:11.283810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.283623 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.291307 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.291280 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.297899 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.297876 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.304623 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.304555 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:11.304623 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.304582 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.309454 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.309436 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:11.311293 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.311279 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.318078 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.318063 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.324246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.324229 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.324509 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.324490 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.324549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.324518 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.324549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.324537 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.324838 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.324814 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.324934 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.324845 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.324934 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.324862 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.325136 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.325118 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.325200 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325143 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.325200 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325166 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.325446 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.325426 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.325494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325453 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.325494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325477 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.325808 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.325785 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.325874 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325814 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.325874 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.325832 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.326098 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.326082 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.326144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326102 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.326144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326116 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.326419 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:00:11.326393 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.326539 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326423 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.326539 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326444 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.326686 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326667 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.326745 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326686 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.326985 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326966 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.327024 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.326986 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.327206 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327190 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.327265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327207 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.327445 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327423 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.327501 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327447 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.327684 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327668 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.327722 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327686 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.327935 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327918 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.327979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.327936 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.328142 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328121 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.328210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328145 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.328368 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328350 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.328415 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328369 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.328582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328565 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.328625 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328582 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.328798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328783 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.328849 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328797 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.329005 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.328987 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.329072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329007 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.329230 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329213 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.329295 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329232 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.329457 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329439 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.329513 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329457 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.329660 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329642 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.329708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329660 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.329894 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329876 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.329952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.329894 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.330106 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330076 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.330106 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330099 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.330291 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330269 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.330349 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330292 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.330536 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330519 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.330583 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330537 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.330780 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330742 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.330780 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330780 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.330999 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.330983 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.331043 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331000 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.331203 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331188 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.331242 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331203 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.331394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331376 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.331439 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331394 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.331563 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331546 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.331607 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331565 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.331774 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331744 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.331817 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.331776 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.332074 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332020 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.332074 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332073 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.332502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332479 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.332502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332501 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.332793 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332722 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.332793 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.332769 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.333063 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333030 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.333063 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333061 2561 scope.go:117] "RemoveContainer" containerID="a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98" Jul 13 21:00:11.333330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333306 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98"} err="failed to get container status \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": rpc error: code = NotFound desc = could not find container \"a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98\": container with ID starting with a19ae82164e563db986495d0a4e1bb18ff47ed83e5d1e88b651bd8e0b827ef98 not found: ID does not exist" Jul 13 21:00:11.333383 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333333 2561 scope.go:117] "RemoveContainer" containerID="f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63" Jul 13 21:00:11.333581 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333563 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63"} err="failed to get container status \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": rpc error: code = NotFound desc = could not find container \"f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63\": container with ID starting with f53144be91ee735d753d9465c3ef36e1c2dbae4300ba7b2a1e70f2d31136cf63 not found: ID does not exist" Jul 13 21:00:11.333633 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333583 2561 scope.go:117] "RemoveContainer" containerID="3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae" Jul 13 21:00:11.333742 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333718 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:11.333856 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333842 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae"} err="failed to get container status \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": rpc error: code = NotFound desc = could not find container \"3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae\": container with ID starting with 3748b4adf874ec6faf51df29ad1cbf67106a3d227c60c5c9f1e66fac075c49ae not found: ID does not exist" Jul 13 21:00:11.333856 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.333856 2561 scope.go:117] "RemoveContainer" containerID="80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b" Jul 13 21:00:11.334077 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334061 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-web" Jul 13 21:00:11.334139 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334086 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-web" Jul 13 21:00:11.334139 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334102 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="thanos-sidecar" Jul 13 21:00:11.334139 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334111 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="thanos-sidecar" Jul 13 21:00:11.334139 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334125 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab791a-f87d-444c-9735-5012f06316f4" containerName="console" Jul 13 21:00:11.334139 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334133 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab791a-f87d-444c-9735-5012f06316f4" containerName="console" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334058 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b"} err="failed to get container status \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": rpc error: code = NotFound desc = could not find container \"80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b\": container with ID starting with 80c2caf0da1a5f9d629e793dc886df1e064d264ddd20c7fbc894a2dc8d62494b not found: ID does not exist" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334153 2561 scope.go:117] "RemoveContainer" containerID="9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334144 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-thanos" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334195 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-thanos" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334212 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="config-reloader" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334222 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="config-reloader" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334234 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="prometheus" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334240 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="prometheus" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334248 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae581f63-2c88-48ea-acda-101e930ba694" containerName="registry" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334255 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae581f63-2c88-48ea-acda-101e930ba694" containerName="registry" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334273 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334281 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334292 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="init-config-reloader" Jul 13 21:00:11.334323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334300 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="init-config-reloader" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334377 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="config-reloader" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334386 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-thanos" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334379 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7"} err="failed to get container status \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": rpc error: code = NotFound desc = could not find container \"9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7\": container with ID starting with 9b3d27ceffbdf7ca19dd18cda8414e18d56713b301e821983253bdd0632d62c7 not found: ID does not exist" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334404 2561 scope.go:117] "RemoveContainer" containerID="d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334393 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="prometheus" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334469 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="thanos-sidecar" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334475 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334482 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae581f63-2c88-48ea-acda-101e930ba694" containerName="registry" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334490 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab791a-f87d-444c-9735-5012f06316f4" containerName="console" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334498 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" containerName="kube-rbac-proxy-web" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334613 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60"} err="failed to get container status \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": rpc error: code = NotFound desc = could not find container \"d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60\": container with ID starting with d22ec53b22de2e641bea51a99a3cfd4eefa62334a5c3bd5829e5a9d8288f6d60 not found: ID does not exist" Jul 13 21:00:11.334808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334626 2561 scope.go:117] "RemoveContainer" containerID="e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119" Jul 13 21:00:11.335151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.334837 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119"} err="failed to get container status \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": rpc error: code = NotFound desc = could not find container \"e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119\": container with ID starting with e42010fe96a6aa4b1e3b9e965ff905914285963963b890bdcc05623a823a6119 not found: ID does not exist" Jul 13 21:00:11.337848 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.337832 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.340796 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340764 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 21:00:11.340796 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 21:00:11.340947 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340774 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 21:00:11.340947 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340820 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 21:00:11.340947 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340936 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 21:00:11.341098 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340965 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 21:00:11.341098 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340970 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xkk8s\"" Jul 13 21:00:11.341098 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.340978 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 21:00:11.341241 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341227 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 21:00:11.341505 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341484 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 21:00:11.341587 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341517 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 21:00:11.341587 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341543 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 21:00:11.341587 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341494 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 21:00:11.341725 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341583 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 21:00:11.341850 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.341837 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bppug4s7a7q4k\"" Jul 13 21:00:11.344864 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.344843 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 21:00:11.346694 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.346675 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 21:00:11.352054 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.352031 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:11.445410 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445371 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445410 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445415 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445519 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445613 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445661 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445684 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445701 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445720 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445816 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d49ls\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-kube-api-access-d49ls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445865 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445880 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.445979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.445927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.546784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.546784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546724 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.546784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546746 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546876 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546917 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.546967 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547010 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547026 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547106 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547189 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547219 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547245 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d49ls\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-kube-api-access-d49ls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547371 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.547976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.547707 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.549740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.548368 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.549740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.548505 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.549740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.549167 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.550458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.550197 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.550458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.550338 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.550458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.550429 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.550884 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.550997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551267 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.551115 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.551767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551911 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.551803 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.551911 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.551772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.552193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.552174 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.552251 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.552201 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.552610 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.552593 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.552978 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.552960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.554061 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.554038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.555522 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.555502 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d49ls\" (UniqueName: \"kubernetes.io/projected/9543bcee-dc6a-4fb5-a9df-0dc3d93109e2-kube-api-access-d49ls\") pod \"prometheus-k8s-0\" (UID: \"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.646950 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.646914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:00:11.783588 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:11.783562 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 21:00:11.785995 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:00:11.785967 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9543bcee_dc6a_4fb5_a9df_0dc3d93109e2.slice/crio-b04b98a13d62bbfac9411108fdc3f29c6e1869eb7f53ff661f829b760e361da8 WatchSource:0}: Error finding container b04b98a13d62bbfac9411108fdc3f29c6e1869eb7f53ff661f829b760e361da8: Status 404 returned error can't find the container with id b04b98a13d62bbfac9411108fdc3f29c6e1869eb7f53ff661f829b760e361da8 Jul 13 21:00:12.279168 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:12.279132 2561 generic.go:356] "Generic (PLEG): container finished" podID="9543bcee-dc6a-4fb5-a9df-0dc3d93109e2" containerID="c60a8d017a40e64199a91763eefdd4672682f903374cfcdc3e679a934f009147" exitCode=0 Jul 13 21:00:12.279362 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:12.279223 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerDied","Data":"c60a8d017a40e64199a91763eefdd4672682f903374cfcdc3e679a934f009147"} Jul 13 21:00:12.279362 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:12.279308 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"b04b98a13d62bbfac9411108fdc3f29c6e1869eb7f53ff661f829b760e361da8"} Jul 13 21:00:12.834132 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:12.834049 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4b049b-ae22-45da-9007-b866cd41f4c1" path="/var/lib/kubelet/pods/ee4b049b-ae22-45da-9007-b866cd41f4c1/volumes" Jul 13 21:00:13.287437 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287401 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"0bfdbca9c7307b43bf0504e16e6cdde464aa1ca1bb09a5ad2df00061346dc226"} Jul 13 21:00:13.287582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287466 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"288e100fa9cfc014af99ba59d27d6cea51514b2f1500dc6ed1ff3ab4e2cf672a"} Jul 13 21:00:13.287582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287494 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"163bfdafc4529a4cc21772fa7265abe49a9b44a36183cae62c94b2c0f43dc95b"} Jul 13 21:00:13.287582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287523 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"01c498526cec8f5a4683f62fc0df86a580d191176fdd51caa3064d8142de310a"} Jul 13 21:00:13.287582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287548 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"ec8450aee5b14e28fd4bbcd3fcb01966cf2773a6d7790b0b50fd7532e357515d"} Jul 13 21:00:13.287582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.287571 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9543bcee-dc6a-4fb5-a9df-0dc3d93109e2","Type":"ContainerStarted","Data":"c05d6cc675fba76b83449f0deb0c9d7cee65c3cb68c97b36b5aa9dbddfc31ed0"} Jul 13 21:00:13.316737 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:13.316676 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.296981043 podStartE2EDuration="2.316655323s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.67428ms" isStatefulPod=false podCreationTimestamp="2026-07-13 21:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:00:13.316373316 +0000 UTC m=+389.045433216" watchObservedRunningTime="2026-07-13 21:00:13.316655323 +0000 UTC m=+389.045715221" Jul 13 21:00:16.648083 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:00:16.648051 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:01:11.648142 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:01:11.648107 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:01:11.663183 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:01:11.663159 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:01:12.522217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:01:12.522191 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 21:03:23.098584 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.098554 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jtgmj"] Jul 13 21:03:23.101631 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.101615 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.104303 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.104275 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 13 21:03:23.104458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.104443 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-llltp\"" Jul 13 21:03:23.105158 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.105140 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 13 21:03:23.114389 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.114371 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jtgmj"] Jul 13 21:03:23.217145 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.217123 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgl9\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-kube-api-access-6dgl9\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.217262 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.217149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-bound-sa-token\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.318394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.318366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgl9\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-kube-api-access-6dgl9\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.318481 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.318398 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-bound-sa-token\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.327708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.327679 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-bound-sa-token\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.327850 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.327767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgl9\" (UniqueName: \"kubernetes.io/projected/9defe187-af0e-4140-9681-219b5fc4ed38-kube-api-access-6dgl9\") pod \"cert-manager-55b4d954d5-jtgmj\" (UID: \"9defe187-af0e-4140-9681-219b5fc4ed38\") " pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.410438 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.410368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jtgmj" Jul 13 21:03:23.550744 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.550713 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jtgmj"] Jul 13 21:03:23.553558 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:03:23.553530 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9defe187_af0e_4140_9681_219b5fc4ed38.slice/crio-4269a52ab5e3e3f087814f1f067ef47af3e94dc6d9ae2ea97983745c9856ed55 WatchSource:0}: Error finding container 4269a52ab5e3e3f087814f1f067ef47af3e94dc6d9ae2ea97983745c9856ed55: Status 404 returned error can't find the container with id 4269a52ab5e3e3f087814f1f067ef47af3e94dc6d9ae2ea97983745c9856ed55 Jul 13 21:03:23.966619 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:23.966578 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jtgmj" event={"ID":"9defe187-af0e-4140-9681-219b5fc4ed38","Type":"ContainerStarted","Data":"4269a52ab5e3e3f087814f1f067ef47af3e94dc6d9ae2ea97983745c9856ed55"} Jul 13 21:03:26.980797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:26.980739 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jtgmj" event={"ID":"9defe187-af0e-4140-9681-219b5fc4ed38","Type":"ContainerStarted","Data":"64dcd875dafa7dd634c6e9612badf05afa562355322685665c0c63e9e4fc392a"} Jul 13 21:03:26.998765 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:26.998708 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jtgmj" podStartSLOduration=0.725685925 podStartE2EDuration="3.998695287s" totalImagesPullingTime="3.273009362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:03:26.997865484 +0000 UTC m=+582.726925408" watchObservedRunningTime="2026-07-13 21:03:26.998695287 +0000 UTC m=+582.727755186" Jul 13 21:03:44.722846 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:44.722816 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:03:44.723551 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:44.723530 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:03:45.932832 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.932799 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv"] Jul 13 21:03:45.936038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.936023 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:45.942708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.942689 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 21:03:45.942999 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.942969 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 21:03:45.943374 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.943357 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 21:03:45.943542 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.943525 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6zt5p\"" Jul 13 21:03:45.944085 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.944070 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 21:03:45.962781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:45.962742 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv"] Jul 13 21:03:46.094293 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.094264 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6g4h\" (UniqueName: \"kubernetes.io/projected/b4c38a8b-204e-4819-aac1-05d02fd8abe2-kube-api-access-n6g4h\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.094458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.094309 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-manifests\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.094458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.094347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-tmp\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.094458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.094407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.094458 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.094451 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.194970 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-manifests\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.195032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-tmp\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195223 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.195067 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195223 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.195085 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195223 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.195108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6g4h\" (UniqueName: \"kubernetes.io/projected/b4c38a8b-204e-4819-aac1-05d02fd8abe2-kube-api-access-n6g4h\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.195392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.195369 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-manifests\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.197280 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.197259 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4c38a8b-204e-4819-aac1-05d02fd8abe2-tmp\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.197628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.197608 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.197680 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.197627 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4c38a8b-204e-4819-aac1-05d02fd8abe2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.215003 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.214979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6g4h\" (UniqueName: \"kubernetes.io/projected/b4c38a8b-204e-4819-aac1-05d02fd8abe2-kube-api-access-n6g4h\") pod \"opendatahub-operator-controller-manager-6fdc557c48-zsjvv\" (UID: \"b4c38a8b-204e-4819-aac1-05d02fd8abe2\") " pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.246492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.246471 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:46.381961 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.381940 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv"] Jul 13 21:03:46.384062 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:03:46.384038 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c38a8b_204e_4819_aac1_05d02fd8abe2.slice/crio-e5a7ce0e0b274fb163012c3b4f8c2a8bbf3d1b0cbba2462f00e9b89014dcaa87 WatchSource:0}: Error finding container e5a7ce0e0b274fb163012c3b4f8c2a8bbf3d1b0cbba2462f00e9b89014dcaa87: Status 404 returned error can't find the container with id e5a7ce0e0b274fb163012c3b4f8c2a8bbf3d1b0cbba2462f00e9b89014dcaa87 Jul 13 21:03:46.385891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:46.385871 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:03:47.051467 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:47.051427 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" event={"ID":"b4c38a8b-204e-4819-aac1-05d02fd8abe2","Type":"ContainerStarted","Data":"e5a7ce0e0b274fb163012c3b4f8c2a8bbf3d1b0cbba2462f00e9b89014dcaa87"} Jul 13 21:03:50.067292 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:50.067255 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" event={"ID":"b4c38a8b-204e-4819-aac1-05d02fd8abe2","Type":"ContainerStarted","Data":"aa1daf69a7733d0b66887df25a48577b7a0a9c86cec2112876c6faca7ab42280"} Jul 13 21:03:51.072334 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:51.072302 2561 generic.go:356] "Generic (PLEG): container finished" podID="b4c38a8b-204e-4819-aac1-05d02fd8abe2" containerID="aa1daf69a7733d0b66887df25a48577b7a0a9c86cec2112876c6faca7ab42280" exitCode=0 Jul 13 21:03:51.072770 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:51.072383 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" event={"ID":"b4c38a8b-204e-4819-aac1-05d02fd8abe2","Type":"ContainerDied","Data":"aa1daf69a7733d0b66887df25a48577b7a0a9c86cec2112876c6faca7ab42280"} Jul 13 21:03:52.077468 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:52.077435 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" event={"ID":"b4c38a8b-204e-4819-aac1-05d02fd8abe2","Type":"ContainerStarted","Data":"02006da382f58c1836149fa58db2a47910121c16333a1a1bd6e26060cdb9f416"} Jul 13 21:03:52.077869 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:52.077550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:03:52.099954 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:03:52.099909 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" podStartSLOduration=2.520194925 podStartE2EDuration="7.099890372s" totalImagesPullingTime="3.511832285s" totalInitContainerRuntime="1.067863162s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:03:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:03:52.098621189 +0000 UTC m=+607.827681093" watchObservedRunningTime="2026-07-13 21:03:52.099890372 +0000 UTC m=+607.828950271" Jul 13 21:04:03.084702 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:03.084671 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fdc557c48-zsjvv" Jul 13 21:04:07.293327 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.293233 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj"] Jul 13 21:04:07.300843 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.300819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.303985 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.303961 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 21:04:07.304151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.303960 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 21:04:07.304938 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.304907 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 21:04:07.305053 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.304956 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 21:04:07.305114 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.305067 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hshbr\"" Jul 13 21:04:07.310471 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.310446 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj"] Jul 13 21:04:07.362854 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.362822 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tmp\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.363004 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.362864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfd69\" (UniqueName: \"kubernetes.io/projected/9028054c-8b61-48a9-92ee-c9d6aaf2add1-kube-api-access-tfd69\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.363004 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.362922 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tls-certs\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.446548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.446510 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6dzzx"] Jul 13 21:04:07.448890 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.448871 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.451306 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.451285 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 21:04:07.451306 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.451294 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mmkdt\"" Jul 13 21:04:07.461250 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.461230 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6dzzx"] Jul 13 21:04:07.464248 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.464225 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfd69\" (UniqueName: \"kubernetes.io/projected/9028054c-8b61-48a9-92ee-c9d6aaf2add1-kube-api-access-tfd69\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.464381 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.464281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tls-certs\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.464381 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.464344 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.464496 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.464378 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-kube-api-access-s7hln\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.464496 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.464410 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tmp\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.466569 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.466538 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tmp\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.466873 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.466857 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9028054c-8b61-48a9-92ee-c9d6aaf2add1-tls-certs\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.478425 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.478404 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfd69\" (UniqueName: \"kubernetes.io/projected/9028054c-8b61-48a9-92ee-c9d6aaf2add1-kube-api-access-tfd69\") pod \"kube-auth-proxy-5bf978c8f5-92btj\" (UID: \"9028054c-8b61-48a9-92ee-c9d6aaf2add1\") " pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.565784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.565685 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.565784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.565723 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-kube-api-access-s7hln\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.565959 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:07.565857 2561 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 21:04:07.565959 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:07.565925 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert podName:24da7a97-e8e1-4bdb-a632-705bfbc59d8b nodeName:}" failed. No retries permitted until 2026-07-13 21:04:08.065906319 +0000 UTC m=+623.794966212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert") pod "odh-model-controller-7998c67dc4-6dzzx" (UID: "24da7a97-e8e1-4bdb-a632-705bfbc59d8b") : secret "odh-model-controller-webhook-cert" not found Jul 13 21:04:07.574668 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.574638 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hln\" (UniqueName: \"kubernetes.io/projected/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-kube-api-access-s7hln\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:07.611480 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.611445 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" Jul 13 21:04:07.748328 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:07.748277 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj"] Jul 13 21:04:07.751231 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:07.751208 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9028054c_8b61_48a9_92ee_c9d6aaf2add1.slice/crio-ed78241fdd2e18a8489c13c6ae91517f121cfda76f9f3b0e4b799bb780f5e38b WatchSource:0}: Error finding container ed78241fdd2e18a8489c13c6ae91517f121cfda76f9f3b0e4b799bb780f5e38b: Status 404 returned error can't find the container with id ed78241fdd2e18a8489c13c6ae91517f121cfda76f9f3b0e4b799bb780f5e38b Jul 13 21:04:08.069243 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.069205 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:08.071549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.071532 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24da7a97-e8e1-4bdb-a632-705bfbc59d8b-cert\") pod \"odh-model-controller-7998c67dc4-6dzzx\" (UID: \"24da7a97-e8e1-4bdb-a632-705bfbc59d8b\") " pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:08.140979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.140943 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" event={"ID":"9028054c-8b61-48a9-92ee-c9d6aaf2add1","Type":"ContainerStarted","Data":"ed78241fdd2e18a8489c13c6ae91517f121cfda76f9f3b0e4b799bb780f5e38b"} Jul 13 21:04:08.359057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.359021 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:08.370051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.370025 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5w8tn"] Jul 13 21:04:08.372783 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.372745 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.375616 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.375434 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 21:04:08.375727 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.375700 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qlcw6\"" Jul 13 21:04:08.384081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.383616 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5w8tn"] Jul 13 21:04:08.472580 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.472518 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k542r\" (UniqueName: \"kubernetes.io/projected/6cb76ffd-f6ec-4931-9f90-6e386b61c177-kube-api-access-k542r\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.472580 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.472568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.544546 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.544517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6dzzx"] Jul 13 21:04:08.547537 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:08.547509 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24da7a97_e8e1_4bdb_a632_705bfbc59d8b.slice/crio-627b23552b0da13b4459dbbd1b30c719a4a7fc7081492ec7ce94aa702f6f24aa WatchSource:0}: Error finding container 627b23552b0da13b4459dbbd1b30c719a4a7fc7081492ec7ce94aa702f6f24aa: Status 404 returned error can't find the container with id 627b23552b0da13b4459dbbd1b30c719a4a7fc7081492ec7ce94aa702f6f24aa Jul 13 21:04:08.573385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.573358 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k542r\" (UniqueName: \"kubernetes.io/projected/6cb76ffd-f6ec-4931-9f90-6e386b61c177-kube-api-access-k542r\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.573516 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.573405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.573621 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:08.573602 2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 21:04:08.573675 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:08.573670 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs podName:6cb76ffd-f6ec-4931-9f90-6e386b61c177 nodeName:}" failed. No retries permitted until 2026-07-13 21:04:09.073648691 +0000 UTC m=+624.802708573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs") pod "model-serving-api-54d89bd5fc-5w8tn" (UID: "6cb76ffd-f6ec-4931-9f90-6e386b61c177") : secret "model-serving-api-tls" not found Jul 13 21:04:08.584456 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.584425 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k542r\" (UniqueName: \"kubernetes.io/projected/6cb76ffd-f6ec-4931-9f90-6e386b61c177-kube-api-access-k542r\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:08.759905 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.759820 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:08.762394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.762368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.765327 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.765301 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 21:04:08.765449 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.765300 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 21:04:08.765740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.765708 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 21:04:08.765964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.765949 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 21:04:08.766497 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.766478 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zk4hq\"" Jul 13 21:04:08.783419 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.783393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:08.875872 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.875832 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876053 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.875890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876053 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.875968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876170 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.876080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876170 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.876141 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.876188 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2d8\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.876875 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.876566 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.977915 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.977878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.977915 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.977920 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2d8\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.977942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.978008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.978038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.978064 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.978128 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.978692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.978668 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.980335 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.980308 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.980523 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.980497 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.980740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.980718 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.980828 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.980767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.986111 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.986090 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:08.986568 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:08.986550 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2d8\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8\") pod \"istiod-openshift-gateway-7574459656-2v77w\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:09.073872 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:09.073782 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:09.078852 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:09.078823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:09.078963 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:09.078942 2561 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 21:04:09.079029 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:09.079022 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs podName:6cb76ffd-f6ec-4931-9f90-6e386b61c177 nodeName:}" failed. No retries permitted until 2026-07-13 21:04:10.079000188 +0000 UTC m=+625.808060068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs") pod "model-serving-api-54d89bd5fc-5w8tn" (UID: "6cb76ffd-f6ec-4931-9f90-6e386b61c177") : secret "model-serving-api-tls" not found Jul 13 21:04:09.147700 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:09.147657 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" event={"ID":"24da7a97-e8e1-4bdb-a632-705bfbc59d8b","Type":"ContainerStarted","Data":"627b23552b0da13b4459dbbd1b30c719a4a7fc7081492ec7ce94aa702f6f24aa"} Jul 13 21:04:09.217735 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:09.217446 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:09.220830 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:09.220797 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be1379e_dfc3_4efa_950d_396f6766fdb2.slice/crio-a0595f4bf363dc647fce8f12d1a272a87d9be70b055a8f6546c6fc8d464ea921 WatchSource:0}: Error finding container a0595f4bf363dc647fce8f12d1a272a87d9be70b055a8f6546c6fc8d464ea921: Status 404 returned error can't find the container with id a0595f4bf363dc647fce8f12d1a272a87d9be70b055a8f6546c6fc8d464ea921 Jul 13 21:04:10.090715 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:10.089930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:10.093069 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:10.093040 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb76ffd-f6ec-4931-9f90-6e386b61c177-tls-certs\") pod \"model-serving-api-54d89bd5fc-5w8tn\" (UID: \"6cb76ffd-f6ec-4931-9f90-6e386b61c177\") " pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:10.154907 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:10.154864 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" event={"ID":"8be1379e-dfc3-4efa-950d-396f6766fdb2","Type":"ContainerStarted","Data":"a0595f4bf363dc647fce8f12d1a272a87d9be70b055a8f6546c6fc8d464ea921"} Jul 13 21:04:10.188479 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:10.188445 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:12.409029 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:12.408998 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5w8tn"] Jul 13 21:04:12.414014 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:12.413981 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb76ffd_f6ec_4931_9f90_6e386b61c177.slice/crio-cbf323c61945b04463cfc2309dbe50d449e333046087b11a3717029502d45f17 WatchSource:0}: Error finding container cbf323c61945b04463cfc2309dbe50d449e333046087b11a3717029502d45f17: Status 404 returned error can't find the container with id cbf323c61945b04463cfc2309dbe50d449e333046087b11a3717029502d45f17 Jul 13 21:04:13.172911 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.172828 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" event={"ID":"9028054c-8b61-48a9-92ee-c9d6aaf2add1","Type":"ContainerStarted","Data":"4e60c9b95fd2ff07b7d135b9ed15c0f1c451b34cab3fc48072eb9bfbe12f9093"} Jul 13 21:04:13.174369 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.174328 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" event={"ID":"6cb76ffd-f6ec-4931-9f90-6e386b61c177","Type":"ContainerStarted","Data":"cbf323c61945b04463cfc2309dbe50d449e333046087b11a3717029502d45f17"} Jul 13 21:04:13.176052 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.176009 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" event={"ID":"24da7a97-e8e1-4bdb-a632-705bfbc59d8b","Type":"ContainerStarted","Data":"7d1136ad52d651c30ab0160de852b6eddfd4d25e528142047c956f897a563314"} Jul 13 21:04:13.176180 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.176160 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:13.192068 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.192001 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bf978c8f5-92btj" podStartSLOduration=1.710868825 podStartE2EDuration="6.191981s" totalImagesPullingTime="4.481112175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:13.189803318 +0000 UTC m=+628.918863242" watchObservedRunningTime="2026-07-13 21:04:13.191981 +0000 UTC m=+628.921040902" Jul 13 21:04:13.205297 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.205214 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" podStartSLOduration=2.46768028 podStartE2EDuration="6.205193381s" totalImagesPullingTime="3.737513101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:13.20442237 +0000 UTC m=+628.933482274" watchObservedRunningTime="2026-07-13 21:04:13.205193381 +0000 UTC m=+628.934253283" Jul 13 21:04:13.690059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.690020 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:04:13.690327 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:13.690116 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:04:14.181615 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:14.181579 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" event={"ID":"8be1379e-dfc3-4efa-950d-396f6766fdb2","Type":"ContainerStarted","Data":"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661"} Jul 13 21:04:14.210814 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:14.210736 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" podStartSLOduration=1.744361154 podStartE2EDuration="6.210715247s" totalImagesPullingTime="4.466354093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:14.208014429 +0000 UTC m=+629.937074756" watchObservedRunningTime="2026-07-13 21:04:14.210715247 +0000 UTC m=+629.939775149" Jul 13 21:04:15.186264 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:15.186229 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:15.188181 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:15.188139 2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-2v77w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 21:04:15.188337 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:15.188211 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 21:04:16.192676 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:16.192632 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" event={"ID":"6cb76ffd-f6ec-4931-9f90-6e386b61c177","Type":"ContainerStarted","Data":"51ace6190dd3b4d3c8c0496f738a087e6a56bf2591c784a3cbb3024e14c5c1d0"} Jul 13 21:04:16.193128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:16.192885 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:16.193655 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:16.193637 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:16.217011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:16.216964 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" podStartSLOduration=5.408446662 podStartE2EDuration="8.216951521s" totalImagesPullingTime="2.808504859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:16.209292052 +0000 UTC m=+631.938351971" watchObservedRunningTime="2026-07-13 21:04:16.216951521 +0000 UTC m=+631.946011421" Jul 13 21:04:18.979299 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.979265 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6"] Jul 13 21:04:18.982239 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.982213 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:18.983704 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.983682 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97"] Jul 13 21:04:18.984656 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.984638 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 21:04:18.984739 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.984683 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-m2wd8\"" Jul 13 21:04:18.987122 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.987105 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:18.989168 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.989149 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dmtgd\"" Jul 13 21:04:18.989498 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.989483 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 21:04:18.993897 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.993877 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6"] Jul 13 21:04:18.996445 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:18.996426 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97"] Jul 13 21:04:19.070402 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.070362 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597fh\" (UniqueName: \"kubernetes.io/projected/b46fdf6d-e08e-4632-a577-015ef20c7b43-kube-api-access-597fh\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.070545 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.070416 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.070545 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.070500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwh2m\" (UniqueName: \"kubernetes.io/projected/42559707-66f4-45dd-87b7-3e65885029e5-kube-api-access-nwh2m\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.070615 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.070557 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46fdf6d-e08e-4632-a577-015ef20c7b43-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.171816 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.171783 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597fh\" (UniqueName: \"kubernetes.io/projected/b46fdf6d-e08e-4632-a577-015ef20c7b43-kube-api-access-597fh\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.171980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.171832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.171980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.171857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwh2m\" (UniqueName: \"kubernetes.io/projected/42559707-66f4-45dd-87b7-3e65885029e5-kube-api-access-nwh2m\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.171980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.171901 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46fdf6d-e08e-4632-a577-015ef20c7b43-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.171980 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:19.171936 2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 21:04:19.172202 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:19.171997 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert podName:42559707-66f4-45dd-87b7-3e65885029e5 nodeName:}" failed. No retries permitted until 2026-07-13 21:04:19.671981411 +0000 UTC m=+635.401041294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert") pod "kserve-controller-manager-d6f4b65b8-wd5g6" (UID: "42559707-66f4-45dd-87b7-3e65885029e5") : secret "kserve-webhook-server-cert" not found Jul 13 21:04:19.175013 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.174989 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b46fdf6d-e08e-4632-a577-015ef20c7b43-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.188790 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.188768 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-597fh\" (UniqueName: \"kubernetes.io/projected/b46fdf6d-e08e-4632-a577-015ef20c7b43-kube-api-access-597fh\") pod \"llmisvc-controller-manager-777c6bd7d5-fbn97\" (UID: \"b46fdf6d-e08e-4632-a577-015ef20c7b43\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.190518 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.190495 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwh2m\" (UniqueName: \"kubernetes.io/projected/42559707-66f4-45dd-87b7-3e65885029e5-kube-api-access-nwh2m\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.301245 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.301155 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:19.429295 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.429234 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97"] Jul 13 21:04:19.431271 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:19.431247 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb46fdf6d_e08e_4632_a577_015ef20c7b43.slice/crio-4d429d8729ae6489899cc36b9c23e13143e4fa6d0552de2edc603f4c4494912d WatchSource:0}: Error finding container 4d429d8729ae6489899cc36b9c23e13143e4fa6d0552de2edc603f4c4494912d: Status 404 returned error can't find the container with id 4d429d8729ae6489899cc36b9c23e13143e4fa6d0552de2edc603f4c4494912d Jul 13 21:04:19.677593 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.677558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.679861 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.679843 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42559707-66f4-45dd-87b7-3e65885029e5-cert\") pod \"kserve-controller-manager-d6f4b65b8-wd5g6\" (UID: \"42559707-66f4-45dd-87b7-3e65885029e5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:19.896487 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:19.896459 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:20.028491 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:20.028471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6"] Jul 13 21:04:20.030351 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:04:20.030327 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42559707_66f4_45dd_87b7_3e65885029e5.slice/crio-c117fd50a94abc1044078b349eef04b680848e4f746784016198f5f5b0d6ebad WatchSource:0}: Error finding container c117fd50a94abc1044078b349eef04b680848e4f746784016198f5f5b0d6ebad: Status 404 returned error can't find the container with id c117fd50a94abc1044078b349eef04b680848e4f746784016198f5f5b0d6ebad Jul 13 21:04:20.210282 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:20.210209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" event={"ID":"42559707-66f4-45dd-87b7-3e65885029e5","Type":"ContainerStarted","Data":"c117fd50a94abc1044078b349eef04b680848e4f746784016198f5f5b0d6ebad"} Jul 13 21:04:20.211318 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:20.211294 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" event={"ID":"b46fdf6d-e08e-4632-a577-015ef20c7b43","Type":"ContainerStarted","Data":"4d429d8729ae6489899cc36b9c23e13143e4fa6d0552de2edc603f4c4494912d"} Jul 13 21:04:21.741719 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:21.741685 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:21.742173 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:21.741939 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerName="discovery" containerID="cri-o://2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661" gracePeriod=30 Jul 13 21:04:22.048238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.048212 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:22.099083 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099053 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099112 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099163 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099201 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099246 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099227 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t2d8\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099451 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099259 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099451 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099322 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert\") pod \"8be1379e-dfc3-4efa-950d-396f6766fdb2\" (UID: \"8be1379e-dfc3-4efa-950d-396f6766fdb2\") " Jul 13 21:04:22.099901 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.099861 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:04:22.102226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102189 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts" (OuterVolumeSpecName: "cacerts") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:04:22.102226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102213 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:04:22.102382 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102318 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs" (OuterVolumeSpecName: "local-certs") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:04:22.102382 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102362 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:04:22.102788 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102737 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:04:22.102788 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.102768 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8" (OuterVolumeSpecName: "kube-api-access-2t2d8") pod "8be1379e-dfc3-4efa-950d-396f6766fdb2" (UID: "8be1379e-dfc3-4efa-950d-396f6766fdb2"). InnerVolumeSpecName "kube-api-access-2t2d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:04:22.200622 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200584 2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-cacerts\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200622 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200614 2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8be1379e-dfc3-4efa-950d-396f6766fdb2-local-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200837 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200629 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2d8\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-kube-api-access-2t2d8\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200837 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200642 2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-kubeconfig\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200837 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200654 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-dns-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200837 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200665 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.200837 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.200676 2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8be1379e-dfc3-4efa-950d-396f6766fdb2-istio-csr-ca-configmap\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:04:22.224696 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.224663 2561 generic.go:356] "Generic (PLEG): container finished" podID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerID="2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661" exitCode=0 Jul 13 21:04:22.224893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.224718 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" Jul 13 21:04:22.224893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.224767 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" event={"ID":"8be1379e-dfc3-4efa-950d-396f6766fdb2","Type":"ContainerDied","Data":"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661"} Jul 13 21:04:22.224893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.224816 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2v77w" event={"ID":"8be1379e-dfc3-4efa-950d-396f6766fdb2","Type":"ContainerDied","Data":"a0595f4bf363dc647fce8f12d1a272a87d9be70b055a8f6546c6fc8d464ea921"} Jul 13 21:04:22.224893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.224835 2561 scope.go:117] "RemoveContainer" containerID="2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661" Jul 13 21:04:22.249191 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.249162 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:22.254210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.254184 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2v77w"] Jul 13 21:04:22.769019 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.768885 2561 scope.go:117] "RemoveContainer" containerID="2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661" Jul 13 21:04:22.769242 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:04:22.769213 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661\": container with ID starting with 2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661 not found: ID does not exist" containerID="2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661" Jul 13 21:04:22.769284 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.769255 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661"} err="failed to get container status \"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661\": rpc error: code = NotFound desc = could not find container \"2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661\": container with ID starting with 2627198bb960efdc3f60d0370303a01015d9b09132311e9faaf98e652324c661 not found: ID does not exist" Jul 13 21:04:22.835417 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:22.835382 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" path="/var/lib/kubelet/pods/8be1379e-dfc3-4efa-950d-396f6766fdb2/volumes" Jul 13 21:04:24.184431 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.184398 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6dzzx" Jul 13 21:04:24.238565 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.238446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" event={"ID":"b46fdf6d-e08e-4632-a577-015ef20c7b43","Type":"ContainerStarted","Data":"8912c1c6c71fa97be1d2fd948146ecc448bdcfa76a00af86b5e36d72020ef3cb"} Jul 13 21:04:24.238565 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.238516 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:24.240143 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.240118 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" event={"ID":"42559707-66f4-45dd-87b7-3e65885029e5","Type":"ContainerStarted","Data":"e53bc3de1b688a4c5d643d9607f3f7119653108472e808aa32906dabd422fd47"} Jul 13 21:04:24.240255 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.240168 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:04:24.259179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.259121 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" podStartSLOduration=2.309329229 podStartE2EDuration="6.259102121s" totalImagesPullingTime="3.949772892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:24.256594634 +0000 UTC m=+639.985654535" watchObservedRunningTime="2026-07-13 21:04:24.259102121 +0000 UTC m=+639.988162022" Jul 13 21:04:24.278841 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:24.278779 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" podStartSLOduration=2.932136489 podStartE2EDuration="6.278762769s" totalImagesPullingTime="3.34662628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:04:24.27640337 +0000 UTC m=+640.005463273" watchObservedRunningTime="2026-07-13 21:04:24.278762769 +0000 UTC m=+640.007822662" Jul 13 21:04:27.201102 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:27.201068 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5w8tn" Jul 13 21:04:55.246010 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:55.245982 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fbn97" Jul 13 21:04:55.248729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:04:55.248711 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-wd5g6" Jul 13 21:05:08.182321 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.182285 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:08.182793 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.182595 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerName="discovery" Jul 13 21:05:08.182793 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.182606 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerName="discovery" Jul 13 21:05:08.182793 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.182658 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be1379e-dfc3-4efa-950d-396f6766fdb2" containerName="discovery" Jul 13 21:05:08.203060 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.203030 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:08.203208 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.203122 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:08.205683 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.205657 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 21:05:08.205840 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.205657 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-ll8pr\"" Jul 13 21:05:08.205840 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.205661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 21:05:08.282708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.282678 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9462\" (UniqueName: \"kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462\") pod \"kuadrant-operator-catalog-snggq\" (UID: \"b2d17665-902a-49d1-868f-97b0f385c293\") " pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:08.383993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.383960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9462\" (UniqueName: \"kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462\") pod \"kuadrant-operator-catalog-snggq\" (UID: \"b2d17665-902a-49d1-868f-97b0f385c293\") " pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:08.392162 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.392127 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9462\" (UniqueName: \"kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462\") pod \"kuadrant-operator-catalog-snggq\" (UID: \"b2d17665-902a-49d1-868f-97b0f385c293\") " pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:08.512871 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.512787 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:08.550099 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.550069 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:08.638187 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.638156 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:08.641205 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:08.641175 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d17665_902a_49d1_868f_97b0f385c293.slice/crio-e6ef9f08b32755254f503dbdc7098a8d8334ab21b6b51236af52096428da775e WatchSource:0}: Error finding container e6ef9f08b32755254f503dbdc7098a8d8334ab21b6b51236af52096428da775e: Status 404 returned error can't find the container with id e6ef9f08b32755254f503dbdc7098a8d8334ab21b6b51236af52096428da775e Jul 13 21:05:08.762104 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.762067 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lmxh"] Jul 13 21:05:08.766600 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.766558 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:08.773471 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.773440 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lmxh"] Jul 13 21:05:08.888427 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.888393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xsj\" (UniqueName: \"kubernetes.io/projected/2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba-kube-api-access-v5xsj\") pod \"kuadrant-operator-catalog-6lmxh\" (UID: \"2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba\") " pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:08.988847 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.988813 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xsj\" (UniqueName: \"kubernetes.io/projected/2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba-kube-api-access-v5xsj\") pod \"kuadrant-operator-catalog-6lmxh\" (UID: \"2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba\") " pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:08.996545 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:08.996518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xsj\" (UniqueName: \"kubernetes.io/projected/2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba-kube-api-access-v5xsj\") pod \"kuadrant-operator-catalog-6lmxh\" (UID: \"2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba\") " pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:09.076986 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:09.076858 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:09.204703 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:09.204673 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6lmxh"] Jul 13 21:05:09.239000 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:09.238964 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6bf8b7_d493_4ba3_a802_66dc68ebe2ba.slice/crio-76cec1d74cb12d6ba887bf3be2cecb213440fa7f6ffb8387cc4e8d899c386979 WatchSource:0}: Error finding container 76cec1d74cb12d6ba887bf3be2cecb213440fa7f6ffb8387cc4e8d899c386979: Status 404 returned error can't find the container with id 76cec1d74cb12d6ba887bf3be2cecb213440fa7f6ffb8387cc4e8d899c386979 Jul 13 21:05:09.427232 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:09.427194 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" event={"ID":"2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba","Type":"ContainerStarted","Data":"76cec1d74cb12d6ba887bf3be2cecb213440fa7f6ffb8387cc4e8d899c386979"} Jul 13 21:05:09.428358 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:09.428334 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-snggq" event={"ID":"b2d17665-902a-49d1-868f-97b0f385c293","Type":"ContainerStarted","Data":"e6ef9f08b32755254f503dbdc7098a8d8334ab21b6b51236af52096428da775e"} Jul 13 21:05:11.438051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.438014 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-snggq" event={"ID":"b2d17665-902a-49d1-868f-97b0f385c293","Type":"ContainerStarted","Data":"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a"} Jul 13 21:05:11.438051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.438027 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-snggq" podUID="b2d17665-902a-49d1-868f-97b0f385c293" containerName="registry-server" containerID="cri-o://950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a" gracePeriod=2 Jul 13 21:05:11.439471 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.439450 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" event={"ID":"2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba","Type":"ContainerStarted","Data":"f6937ab684442aa215a6788ba3ae50c7549ab08efe2917681903d54b8010d236"} Jul 13 21:05:11.454078 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.454030 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-snggq" podStartSLOduration=1.355744525 podStartE2EDuration="3.454012399s" totalImagesPullingTime="2.098267874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:11.451970204 +0000 UTC m=+687.181030107" watchObservedRunningTime="2026-07-13 21:05:11.454012399 +0000 UTC m=+687.183072300" Jul 13 21:05:11.466635 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.466592 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" podStartSLOduration=1.965367048 podStartE2EDuration="3.466577335s" totalImagesPullingTime="1.501210287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:11.46593469 +0000 UTC m=+687.194994590" watchObservedRunningTime="2026-07-13 21:05:11.466577335 +0000 UTC m=+687.195637236" Jul 13 21:05:11.682776 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.682739 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:11.815166 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.815086 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9462\" (UniqueName: \"kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462\") pod \"b2d17665-902a-49d1-868f-97b0f385c293\" (UID: \"b2d17665-902a-49d1-868f-97b0f385c293\") " Jul 13 21:05:11.817160 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.817137 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462" (OuterVolumeSpecName: "kube-api-access-f9462") pod "b2d17665-902a-49d1-868f-97b0f385c293" (UID: "b2d17665-902a-49d1-868f-97b0f385c293"). InnerVolumeSpecName "kube-api-access-f9462". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:05:11.915931 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:11.915899 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9462\" (UniqueName: \"kubernetes.io/projected/b2d17665-902a-49d1-868f-97b0f385c293-kube-api-access-f9462\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:05:12.445341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.445306 2561 generic.go:356] "Generic (PLEG): container finished" podID="b2d17665-902a-49d1-868f-97b0f385c293" containerID="950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a" exitCode=0 Jul 13 21:05:12.445771 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.445353 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-snggq" Jul 13 21:05:12.445771 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.445391 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-snggq" event={"ID":"b2d17665-902a-49d1-868f-97b0f385c293","Type":"ContainerDied","Data":"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a"} Jul 13 21:05:12.445771 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.445441 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-snggq" event={"ID":"b2d17665-902a-49d1-868f-97b0f385c293","Type":"ContainerDied","Data":"e6ef9f08b32755254f503dbdc7098a8d8334ab21b6b51236af52096428da775e"} Jul 13 21:05:12.445771 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.445460 2561 scope.go:117] "RemoveContainer" containerID="950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a" Jul 13 21:05:12.454545 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.454519 2561 scope.go:117] "RemoveContainer" containerID="950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a" Jul 13 21:05:12.454804 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:05:12.454787 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a\": container with ID starting with 950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a not found: ID does not exist" containerID="950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a" Jul 13 21:05:12.454869 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.454813 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a"} err="failed to get container status \"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a\": rpc error: code = NotFound desc = could not find container \"950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a\": container with ID starting with 950722911e89125029a09bf2dbff8d1c8ecacfef56a6a3a59245d497c38d129a not found: ID does not exist" Jul 13 21:05:12.465551 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.465531 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:12.469232 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.469212 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-snggq"] Jul 13 21:05:12.834196 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:12.834115 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d17665-902a-49d1-868f-97b0f385c293" path="/var/lib/kubelet/pods/b2d17665-902a-49d1-868f-97b0f385c293/volumes" Jul 13 21:05:19.077453 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:19.077408 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:19.077998 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:19.077471 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:19.099829 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:19.099802 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:19.498075 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:19.498046 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6lmxh" Jul 13 21:05:20.413791 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.413733 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ccc957564-4s5rt"] Jul 13 21:05:20.414184 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.414083 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d17665-902a-49d1-868f-97b0f385c293" containerName="registry-server" Jul 13 21:05:20.414184 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.414094 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d17665-902a-49d1-868f-97b0f385c293" containerName="registry-server" Jul 13 21:05:20.414184 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.414152 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d17665-902a-49d1-868f-97b0f385c293" containerName="registry-server" Jul 13 21:05:20.418230 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.418212 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.425212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425043 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 21:05:20.425212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425043 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 21:05:20.425212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425071 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 21:05:20.425212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425103 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 21:05:20.425212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425167 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r8vlv\"" Jul 13 21:05:20.425536 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.425372 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 21:05:20.429148 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.429124 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 21:05:20.429613 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.429589 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccc957564-4s5rt"] Jul 13 21:05:20.493187 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493156 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493187 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493190 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-trusted-ca-bundle\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493219 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-oauth-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrs8\" (UniqueName: \"kubernetes.io/projected/ed09fb08-a445-40a1-a1fa-e56feda559f7-kube-api-access-pcrs8\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-oauth-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.493392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.493364 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-service-ca\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594532 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594491 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594555 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-trusted-ca-bundle\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-oauth-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrs8\" (UniqueName: \"kubernetes.io/projected/ed09fb08-a445-40a1-a1fa-e56feda559f7-kube-api-access-pcrs8\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-oauth-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.594705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.594688 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-service-ca\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.595421 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.595390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-oauth-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.595512 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.595500 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.595560 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.595539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-service-ca\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.595675 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.595653 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed09fb08-a445-40a1-a1fa-e56feda559f7-trusted-ca-bundle\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.597157 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.597136 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-serving-cert\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.597244 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.597162 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed09fb08-a445-40a1-a1fa-e56feda559f7-console-oauth-config\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.602079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.602055 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrs8\" (UniqueName: \"kubernetes.io/projected/ed09fb08-a445-40a1-a1fa-e56feda559f7-kube-api-access-pcrs8\") pod \"console-ccc957564-4s5rt\" (UID: \"ed09fb08-a445-40a1-a1fa-e56feda559f7\") " pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.733963 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.733871 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:20.870296 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:20.870262 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccc957564-4s5rt"] Jul 13 21:05:20.872833 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:20.872799 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded09fb08_a445_40a1_a1fa_e56feda559f7.slice/crio-d1232fcf65a4a7d7814cb6901c01132d0849d3e59b49b529dfa42c76cfb813c6 WatchSource:0}: Error finding container d1232fcf65a4a7d7814cb6901c01132d0849d3e59b49b529dfa42c76cfb813c6: Status 404 returned error can't find the container with id d1232fcf65a4a7d7814cb6901c01132d0849d3e59b49b529dfa42c76cfb813c6 Jul 13 21:05:21.485808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:21.485773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccc957564-4s5rt" event={"ID":"ed09fb08-a445-40a1-a1fa-e56feda559f7","Type":"ContainerStarted","Data":"c8cf562cc2561b8dabf0337d74c671c8f0d3ddc7a7023c95ce3c1c7fab2be0f1"} Jul 13 21:05:21.486193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:21.485826 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccc957564-4s5rt" event={"ID":"ed09fb08-a445-40a1-a1fa-e56feda559f7","Type":"ContainerStarted","Data":"d1232fcf65a4a7d7814cb6901c01132d0849d3e59b49b529dfa42c76cfb813c6"} Jul 13 21:05:21.504391 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:21.504341 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ccc957564-4s5rt" podStartSLOduration=1.5043263169999999 podStartE2EDuration="1.504326317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:21.502441969 +0000 UTC m=+697.231501869" watchObservedRunningTime="2026-07-13 21:05:21.504326317 +0000 UTC m=+697.233386217" Jul 13 21:05:30.734807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:30.734760 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:30.735230 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:30.734831 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:30.739517 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:30.739493 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:31.531857 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:31.531829 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ccc957564-4s5rt" Jul 13 21:05:40.525314 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.525232 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj"] Jul 13 21:05:40.528710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.528691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:40.531201 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.531182 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-28lfj\"" Jul 13 21:05:40.539473 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.539453 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj"] Jul 13 21:05:40.557585 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.557560 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4vs\" (UniqueName: \"kubernetes.io/projected/44caa46c-8b5f-4304-be6a-83cd9b06cf23-kube-api-access-wb4vs\") pod \"authorino-operator-75cc8d5cb7-qk4xj\" (UID: \"44caa46c-8b5f-4304-be6a-83cd9b06cf23\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:40.658038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.658013 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4vs\" (UniqueName: \"kubernetes.io/projected/44caa46c-8b5f-4304-be6a-83cd9b06cf23-kube-api-access-wb4vs\") pod \"authorino-operator-75cc8d5cb7-qk4xj\" (UID: \"44caa46c-8b5f-4304-be6a-83cd9b06cf23\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:40.670729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.670701 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4vs\" (UniqueName: \"kubernetes.io/projected/44caa46c-8b5f-4304-be6a-83cd9b06cf23-kube-api-access-wb4vs\") pod \"authorino-operator-75cc8d5cb7-qk4xj\" (UID: \"44caa46c-8b5f-4304-be6a-83cd9b06cf23\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:40.839305 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.839241 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:40.963600 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:40.963576 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj"] Jul 13 21:05:40.965764 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:40.965719 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44caa46c_8b5f_4304_be6a_83cd9b06cf23.slice/crio-d2322884c348291794af3d1107119a0f0af3771217a02546f4fdc969bf49ab52 WatchSource:0}: Error finding container d2322884c348291794af3d1107119a0f0af3771217a02546f4fdc969bf49ab52: Status 404 returned error can't find the container with id d2322884c348291794af3d1107119a0f0af3771217a02546f4fdc969bf49ab52 Jul 13 21:05:41.570604 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:41.570567 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" event={"ID":"44caa46c-8b5f-4304-be6a-83cd9b06cf23","Type":"ContainerStarted","Data":"d2322884c348291794af3d1107119a0f0af3771217a02546f4fdc969bf49ab52"} Jul 13 21:05:43.582549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.582447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" event={"ID":"44caa46c-8b5f-4304-be6a-83cd9b06cf23","Type":"ContainerStarted","Data":"095dca980fda6e6c233f5be31ad0ebd20da60d493a9ca4cb9ea59246b52d3298"} Jul 13 21:05:43.582942 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.582593 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:43.599888 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.599841 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" podStartSLOduration=1.316483582 podStartE2EDuration="3.59982617s" totalImagesPullingTime="2.283342588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:43.598262135 +0000 UTC m=+719.327322048" watchObservedRunningTime="2026-07-13 21:05:43.59982617 +0000 UTC m=+719.328886117" Jul 13 21:05:43.751604 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.751567 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5"] Jul 13 21:05:43.754874 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.754854 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:43.757399 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.757380 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 21:05:43.757399 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.757390 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-t6vqf\"" Jul 13 21:05:43.767569 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.767547 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5"] Jul 13 21:05:43.782412 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.782389 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wc9\" (UniqueName: \"kubernetes.io/projected/74772e18-ece0-48d3-adb1-af8b082f1dbc-kube-api-access-l9wc9\") pod \"dns-operator-controller-manager-864dd78c4c-cs4g5\" (UID: \"74772e18-ece0-48d3-adb1-af8b082f1dbc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:43.882713 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.882686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wc9\" (UniqueName: \"kubernetes.io/projected/74772e18-ece0-48d3-adb1-af8b082f1dbc-kube-api-access-l9wc9\") pod \"dns-operator-controller-manager-864dd78c4c-cs4g5\" (UID: \"74772e18-ece0-48d3-adb1-af8b082f1dbc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:43.896454 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:43.896431 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wc9\" (UniqueName: \"kubernetes.io/projected/74772e18-ece0-48d3-adb1-af8b082f1dbc-kube-api-access-l9wc9\") pod \"dns-operator-controller-manager-864dd78c4c-cs4g5\" (UID: \"74772e18-ece0-48d3-adb1-af8b082f1dbc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:44.065415 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:44.065386 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:44.205000 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:44.204964 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5"] Jul 13 21:05:44.207156 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:44.207131 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74772e18_ece0_48d3_adb1_af8b082f1dbc.slice/crio-3806aa14d9cf5daad675c579ab2e5aac086943a779ccbf08f9e3ddd54b6c3c54 WatchSource:0}: Error finding container 3806aa14d9cf5daad675c579ab2e5aac086943a779ccbf08f9e3ddd54b6c3c54: Status 404 returned error can't find the container with id 3806aa14d9cf5daad675c579ab2e5aac086943a779ccbf08f9e3ddd54b6c3c54 Jul 13 21:05:44.588881 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:44.588801 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" event={"ID":"74772e18-ece0-48d3-adb1-af8b082f1dbc","Type":"ContainerStarted","Data":"3806aa14d9cf5daad675c579ab2e5aac086943a779ccbf08f9e3ddd54b6c3c54"} Jul 13 21:05:46.379665 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.379640 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 21:05:46.455056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.455035 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg"] Jul 13 21:05:46.458255 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.458239 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:46.460702 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.460684 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cr4rk\"" Jul 13 21:05:46.471890 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.471862 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg"] Jul 13 21:05:46.500841 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.500805 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xjb5\" (UniqueName: \"kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5\") pod \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" (UID: \"30c8afbe-1df1-4dee-853c-5ce447e40c4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:46.599729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.599689 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" event={"ID":"74772e18-ece0-48d3-adb1-af8b082f1dbc","Type":"ContainerStarted","Data":"13dd00702bcae03ff11b9fd3362408ad18b995008038e45abfb6820bc7d8afa1"} Jul 13 21:05:46.599911 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.599822 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:46.601221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.601197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xjb5\" (UniqueName: \"kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5\") pod \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" (UID: \"30c8afbe-1df1-4dee-853c-5ce447e40c4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:46.609181 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.609161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xjb5\" (UniqueName: \"kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5\") pod \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" (UID: \"30c8afbe-1df1-4dee-853c-5ce447e40c4c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:46.616790 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.616725 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" podStartSLOduration=1.448886555 podStartE2EDuration="3.616711544s" totalImagesPullingTime="2.167824989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:46.615054972 +0000 UTC m=+722.344114895" watchObservedRunningTime="2026-07-13 21:05:46.616711544 +0000 UTC m=+722.345771445" Jul 13 21:05:46.769182 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.769096 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:46.903671 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:46.903646 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg"] Jul 13 21:05:46.905915 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:46.905880 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c8afbe_1df1_4dee_853c_5ce447e40c4c.slice/crio-026d3af53626b6d4cdf4be630f8dd250bfd4d22bfda51029561830f6f1a877c8 WatchSource:0}: Error finding container 026d3af53626b6d4cdf4be630f8dd250bfd4d22bfda51029561830f6f1a877c8: Status 404 returned error can't find the container with id 026d3af53626b6d4cdf4be630f8dd250bfd4d22bfda51029561830f6f1a877c8 Jul 13 21:05:47.610023 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:47.609990 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" event={"ID":"30c8afbe-1df1-4dee-853c-5ce447e40c4c","Type":"ContainerStarted","Data":"026d3af53626b6d4cdf4be630f8dd250bfd4d22bfda51029561830f6f1a877c8"} Jul 13 21:05:49.620937 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:49.620897 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" event={"ID":"30c8afbe-1df1-4dee-853c-5ce447e40c4c","Type":"ContainerStarted","Data":"6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568"} Jul 13 21:05:49.621375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:49.620978 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:05:49.643273 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:49.643228 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" podStartSLOduration=1.893429968 podStartE2EDuration="3.643214133s" totalImagesPullingTime="1.749784165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:49.640745374 +0000 UTC m=+725.369805286" watchObservedRunningTime="2026-07-13 21:05:49.643214133 +0000 UTC m=+725.372274033" Jul 13 21:05:51.000463 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.000429 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z"] Jul 13 21:05:51.003648 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.003632 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.007172 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.007145 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nk4b5\"" Jul 13 21:05:51.017584 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.017562 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z"] Jul 13 21:05:51.038598 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.038574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkqv\" (UniqueName: \"kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.038761 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.038624 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.139710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.139684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkqv\" (UniqueName: \"kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.139832 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.139729 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.140060 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.140044 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.147821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.147796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkqv\" (UniqueName: \"kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.314342 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.314277 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:51.446632 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.446606 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z"] Jul 13 21:05:51.449115 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:05:51.449090 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639c6e92_ed03_44c0_bce9_d528f20e4482.slice/crio-38f4a89565c16302d51ed0fb8d6461ca5922e9ae7eb470294b92956ba1ef4112 WatchSource:0}: Error finding container 38f4a89565c16302d51ed0fb8d6461ca5922e9ae7eb470294b92956ba1ef4112: Status 404 returned error can't find the container with id 38f4a89565c16302d51ed0fb8d6461ca5922e9ae7eb470294b92956ba1ef4112 Jul 13 21:05:51.631188 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:51.631154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" event={"ID":"639c6e92-ed03-44c0-bce9-d528f20e4482","Type":"ContainerStarted","Data":"38f4a89565c16302d51ed0fb8d6461ca5922e9ae7eb470294b92956ba1ef4112"} Jul 13 21:05:54.590862 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:54.590831 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qk4xj" Jul 13 21:05:57.612470 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:57.612438 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cs4g5" Jul 13 21:05:58.668020 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:58.667984 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" event={"ID":"639c6e92-ed03-44c0-bce9-d528f20e4482","Type":"ContainerStarted","Data":"8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313"} Jul 13 21:05:58.668385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:58.668076 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:05:58.690108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:05:58.690001 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" podStartSLOduration=1.927545951 podStartE2EDuration="8.689980846s" totalImagesPullingTime="6.762434895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:05:58.688148084 +0000 UTC m=+734.417208002" watchObservedRunningTime="2026-07-13 21:05:58.689980846 +0000 UTC m=+734.419040747" Jul 13 21:06:00.627131 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:00.627103 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:09.675827 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:09.675792 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:10.621337 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.621305 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85"] Jul 13 21:06:10.624590 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.624571 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.640087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.640057 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85"] Jul 13 21:06:10.704018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.703983 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.704383 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.704036 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbzg\" (UniqueName: \"kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.804533 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.804491 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.804703 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.804564 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbzg\" (UniqueName: \"kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.804918 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.804894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.815545 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.815517 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbzg\" (UniqueName: \"kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:10.934721 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:10.934645 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:11.065056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.065024 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85"] Jul 13 21:06:11.068979 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:11.068953 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da01438_c653_4fae_b4a8_a6cff51ca7d8.slice/crio-0150451ecb52289cdc1a624f77d5a25f3cb6e4d17439b540dbd654bb1893b8ad WatchSource:0}: Error finding container 0150451ecb52289cdc1a624f77d5a25f3cb6e4d17439b540dbd654bb1893b8ad: Status 404 returned error can't find the container with id 0150451ecb52289cdc1a624f77d5a25f3cb6e4d17439b540dbd654bb1893b8ad Jul 13 21:06:11.380532 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.380497 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85"] Jul 13 21:06:11.390654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.390625 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85"] Jul 13 21:06:11.408387 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.408362 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z"] Jul 13 21:06:11.408631 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.408569 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" containerName="manager" containerID="cri-o://8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313" gracePeriod=2 Jul 13 21:06:11.435731 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.435696 2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"639c6e92-ed03-44c0-bce9-d528f20e4482\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-5n75z\": pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.443595 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.443569 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z"] Jul 13 21:06:11.455194 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.455151 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg"] Jul 13 21:06:11.455628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.455412 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" containerName="manager" containerID="cri-o://6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568" gracePeriod=2 Jul 13 21:06:11.457784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.457760 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458273 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" containerName="manager" Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458297 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" containerName="manager" Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458328 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" containerName="manager" Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458337 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" containerName="manager" Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458422 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" containerName="manager" Jul 13 21:06:11.459087 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.458438 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" containerName="manager" Jul 13 21:06:11.459484 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.459214 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.461292 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.461260 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.462287 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.462267 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:11.462394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.462384 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.465429 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.465401 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.466077 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.466060 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg"] Jul 13 21:06:11.466151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.466138 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.493525 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.493497 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:11.495971 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.495946 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:11.498152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.498130 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb"] Jul 13 21:06:11.498650 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.498632 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" containerName="manager" Jul 13 21:06:11.498692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.498656 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" containerName="manager" Jul 13 21:06:11.498792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.498779 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" containerName="manager" Jul 13 21:06:11.502142 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.502123 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:11.507869 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.507846 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb"] Jul 13 21:06:11.511617 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.511590 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.511735 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.511683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtslx\" (UniqueName: \"kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.512523 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.512488 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.514719 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.514688 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.564374 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.564339 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.612333 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612302 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5jh\" (UniqueName: \"kubernetes.io/projected/eb0d2789-0097-416c-b489-9d0ae55403c8-kube-api-access-wj5jh\") pod \"limitador-operator-controller-manager-859d7b78bd-zjvfb\" (UID: \"eb0d2789-0097-416c-b489-9d0ae55403c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:11.612484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612343 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.612484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612369 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtslx\" (UniqueName: \"kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.612484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.612484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612415 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.612902 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.612878 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.622160 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.622139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtslx\" (UniqueName: \"kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l57pf\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.661734 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.661709 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.664087 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.664058 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.666028 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.666002 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.694265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.694242 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.696437 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.696410 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.698184 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.698164 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.713494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.713471 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhkqv\" (UniqueName: \"kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv\") pod \"639c6e92-ed03-44c0-bce9-d528f20e4482\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " Jul 13 21:06:11.713806 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.713504 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume\") pod \"639c6e92-ed03-44c0-bce9-d528f20e4482\" (UID: \"639c6e92-ed03-44c0-bce9-d528f20e4482\") " Jul 13 21:06:11.713806 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.713687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5jh\" (UniqueName: \"kubernetes.io/projected/eb0d2789-0097-416c-b489-9d0ae55403c8-kube-api-access-wj5jh\") pod \"limitador-operator-controller-manager-859d7b78bd-zjvfb\" (UID: \"eb0d2789-0097-416c-b489-9d0ae55403c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:11.713806 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.713728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.713977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.713926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.714073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.714055 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "639c6e92-ed03-44c0-bce9-d528f20e4482" (UID: "639c6e92-ed03-44c0-bce9-d528f20e4482"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:06:11.714314 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.714293 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.715606 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.715589 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv" (OuterVolumeSpecName: "kube-api-access-qhkqv") pod "639c6e92-ed03-44c0-bce9-d528f20e4482" (UID: "639c6e92-ed03-44c0-bce9-d528f20e4482"). InnerVolumeSpecName "kube-api-access-qhkqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:06:11.723695 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.723672 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5hzcl\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.724021 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.724001 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5jh\" (UniqueName: \"kubernetes.io/projected/eb0d2789-0097-416c-b489-9d0ae55403c8-kube-api-access-wj5jh\") pod \"limitador-operator-controller-manager-859d7b78bd-zjvfb\" (UID: \"eb0d2789-0097-416c-b489-9d0ae55403c8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:11.733692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.733666 2561 generic.go:356] "Generic (PLEG): container finished" podID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" containerID="6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568" exitCode=0 Jul 13 21:06:11.733789 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.733707 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.733789 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.733761 2561 scope.go:117] "RemoveContainer" containerID="6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568" Jul 13 21:06:11.735067 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.735042 2561 generic.go:356] "Generic (PLEG): container finished" podID="639c6e92-ed03-44c0-bce9-d528f20e4482" containerID="8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313" exitCode=0 Jul 13 21:06:11.735169 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.735071 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.736101 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.736075 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.737108 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.737085 2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" container="manager" Jul 13 21:06:11.738093 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.738052 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.739953 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.739919 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.741872 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.741848 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.742853 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.742827 2561 scope.go:117] "RemoveContainer" containerID="6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568" Jul 13 21:06:11.743131 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.743100 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568\": container with ID starting with 6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568 not found: ID does not exist" containerID="6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568" Jul 13 21:06:11.743191 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.743135 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568"} err="failed to get container status \"6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568\": rpc error: code = NotFound desc = could not find container \"6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568\": container with ID starting with 6d8d204d6bf7609743af216df710cd67e7ccb274e258d448e373d6609bedc568 not found: ID does not exist" Jul 13 21:06:11.743191 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.743157 2561 scope.go:117] "RemoveContainer" containerID="8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313" Jul 13 21:06:11.743742 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.743717 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:11.745961 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.745939 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:11.747896 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.747874 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:11.749795 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.749772 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:11.751444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.751430 2561 scope.go:117] "RemoveContainer" containerID="8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313" Jul 13 21:06:11.751720 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:11.751698 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313\": container with ID starting with 8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313 not found: ID does not exist" containerID="8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313" Jul 13 21:06:11.751827 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.751726 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313"} err="failed to get container status \"8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313\": rpc error: code = NotFound desc = could not find container \"8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313\": container with ID starting with 8436e21816623d78c07984c77d8312db5896eb274a89f27a1f5bd7cfd0f10313 not found: ID does not exist" Jul 13 21:06:11.815004 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.814974 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xjb5\" (UniqueName: \"kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5\") pod \"30c8afbe-1df1-4dee-853c-5ce447e40c4c\" (UID: \"30c8afbe-1df1-4dee-853c-5ce447e40c4c\") " Jul 13 21:06:11.815168 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.815154 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkqv\" (UniqueName: \"kubernetes.io/projected/639c6e92-ed03-44c0-bce9-d528f20e4482-kube-api-access-qhkqv\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:11.815235 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.815171 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/639c6e92-ed03-44c0-bce9-d528f20e4482-extensions-socket-volume\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:11.816923 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.816899 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5" (OuterVolumeSpecName: "kube-api-access-2xjb5") pod "30c8afbe-1df1-4dee-853c-5ce447e40c4c" (UID: "30c8afbe-1df1-4dee-853c-5ce447e40c4c"). InnerVolumeSpecName "kube-api-access-2xjb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:06:11.853278 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.853251 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:11.863309 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.863278 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:11.872348 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.872328 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:11.915870 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:11.915824 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xjb5\" (UniqueName: \"kubernetes.io/projected/30c8afbe-1df1-4dee-853c-5ce447e40c4c-kube-api-access-2xjb5\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:12.021058 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.021026 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:12.024759 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:12.024722 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eedecb_9b6b_4866_9e40_35098846447e.slice/crio-26f07217f97555e455c4ef10f0e40335f0037816f5d64bce06af73e8b4e61d4a WatchSource:0}: Error finding container 26f07217f97555e455c4ef10f0e40335f0037816f5d64bce06af73e8b4e61d4a: Status 404 returned error can't find the container with id 26f07217f97555e455c4ef10f0e40335f0037816f5d64bce06af73e8b4e61d4a Jul 13 21:06:12.033711 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.033686 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:12.037634 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:12.037612 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ea7624_43ec_410b_80aa_77e4423d3883.slice/crio-1a1eccdcc7707e83f4061a6ed4d1b8b1383cffd784a0c9547b17d11e3adc98c1 WatchSource:0}: Error finding container 1a1eccdcc7707e83f4061a6ed4d1b8b1383cffd784a0c9547b17d11e3adc98c1: Status 404 returned error can't find the container with id 1a1eccdcc7707e83f4061a6ed4d1b8b1383cffd784a0c9547b17d11e3adc98c1 Jul 13 21:06:12.045068 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.045037 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:12.047036 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.047009 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:12.048985 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.048956 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:12.060167 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.060143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb"] Jul 13 21:06:12.063735 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:12.063672 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0d2789_0097_416c_b489_9d0ae55403c8.slice/crio-df3301b19fa0257cfcd4cdeb0667c9c88e3a720f77c1158c8c3950b7734a3122 WatchSource:0}: Error finding container df3301b19fa0257cfcd4cdeb0667c9c88e3a720f77c1158c8c3950b7734a3122: Status 404 returned error can't find the container with id df3301b19fa0257cfcd4cdeb0667c9c88e3a720f77c1158c8c3950b7734a3122 Jul 13 21:06:12.745721 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.745654 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" event={"ID":"f4ea7624-43ec-410b-80aa-77e4423d3883","Type":"ContainerStarted","Data":"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83"} Jul 13 21:06:12.745721 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.745714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" event={"ID":"f4ea7624-43ec-410b-80aa-77e4423d3883","Type":"ContainerStarted","Data":"1a1eccdcc7707e83f4061a6ed4d1b8b1383cffd784a0c9547b17d11e3adc98c1"} Jul 13 21:06:12.746201 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.745821 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:12.747374 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.747347 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" event={"ID":"53eedecb-9b6b-4866-9e40-35098846447e","Type":"ContainerStarted","Data":"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528"} Jul 13 21:06:12.747509 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.747389 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" event={"ID":"53eedecb-9b6b-4866-9e40-35098846447e","Type":"ContainerStarted","Data":"26f07217f97555e455c4ef10f0e40335f0037816f5d64bce06af73e8b4e61d4a"} Jul 13 21:06:12.747509 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.747423 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:12.749052 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.749025 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" containerName="manager" containerID="cri-o://f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4" gracePeriod=2 Jul 13 21:06:12.750502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.750477 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" event={"ID":"eb0d2789-0097-416c-b489-9d0ae55403c8","Type":"ContainerStarted","Data":"08df85c84f5730d46a63990920f49a0deb28e60149cc920fc089c7b897d8e673"} Jul 13 21:06:12.750616 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.750534 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:12.750616 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.750563 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" event={"ID":"eb0d2789-0097-416c-b489-9d0ae55403c8","Type":"ContainerStarted","Data":"df3301b19fa0257cfcd4cdeb0667c9c88e3a720f77c1158c8c3950b7734a3122"} Jul 13 21:06:12.770619 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.770576 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" podStartSLOduration=1.770562983 podStartE2EDuration="1.770562983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:12.770426391 +0000 UTC m=+748.499486310" watchObservedRunningTime="2026-07-13 21:06:12.770562983 +0000 UTC m=+748.499622885" Jul 13 21:06:12.772894 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.772870 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:12.774689 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.774668 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:12.776809 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.776788 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:12.806945 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.806898 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" podStartSLOduration=1.806886962 podStartE2EDuration="1.806886962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:12.805530238 +0000 UTC m=+748.534590161" watchObservedRunningTime="2026-07-13 21:06:12.806886962 +0000 UTC m=+748.535946927" Jul 13 21:06:12.829242 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.829198 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" podStartSLOduration=1.829179428 podStartE2EDuration="1.829179428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:12.827633982 +0000 UTC m=+748.556693927" watchObservedRunningTime="2026-07-13 21:06:12.829179428 +0000 UTC m=+748.558239328" Jul 13 21:06:12.830322 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.830291 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5n75z\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5n75z" Jul 13 21:06:12.832197 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.832173 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:12.833986 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:12.833963 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pvvhg\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pvvhg" Jul 13 21:06:12.834287 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.834264 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c8afbe-1df1-4dee-853c-5ce447e40c4c" path="/var/lib/kubelet/pods/30c8afbe-1df1-4dee-853c-5ce447e40c4c/volumes" Jul 13 21:06:12.834656 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.834642 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="639c6e92-ed03-44c0-bce9-d528f20e4482" path="/var/lib/kubelet/pods/639c6e92-ed03-44c0-bce9-d528f20e4482/volumes" Jul 13 21:06:12.996090 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:12.996030 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:13.126189 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.126164 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume\") pod \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " Jul 13 21:06:13.126343 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.126247 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkbzg\" (UniqueName: \"kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg\") pod \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\" (UID: \"5da01438-c653-4fae-b4a8-a6cff51ca7d8\") " Jul 13 21:06:13.126572 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.126543 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5da01438-c653-4fae-b4a8-a6cff51ca7d8" (UID: "5da01438-c653-4fae-b4a8-a6cff51ca7d8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:06:13.128155 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.128133 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg" (OuterVolumeSpecName: "kube-api-access-fkbzg") pod "5da01438-c653-4fae-b4a8-a6cff51ca7d8" (UID: "5da01438-c653-4fae-b4a8-a6cff51ca7d8"). InnerVolumeSpecName "kube-api-access-fkbzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:06:13.226913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.226884 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da01438-c653-4fae-b4a8-a6cff51ca7d8-extensions-socket-volume\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:13.226913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.226906 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbzg\" (UniqueName: \"kubernetes.io/projected/5da01438-c653-4fae-b4a8-a6cff51ca7d8-kube-api-access-fkbzg\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:13.755899 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.755861 2561 generic.go:356] "Generic (PLEG): container finished" podID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" containerID="f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4" exitCode=0 Jul 13 21:06:13.756357 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.755914 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:13.756357 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.755962 2561 scope.go:117] "RemoveContainer" containerID="f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4" Jul 13 21:06:13.758543 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:13.758511 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:13.765009 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.764991 2561 scope.go:117] "RemoveContainer" containerID="f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4" Jul 13 21:06:13.765243 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:13.765225 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4\": container with ID starting with f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4 not found: ID does not exist" containerID="f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4" Jul 13 21:06:13.765304 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:13.765253 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4"} err="failed to get container status \"f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4\": rpc error: code = NotFound desc = could not find container \"f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4\": container with ID starting with f079b0f0cd98861d120b43afb29475b0b8c9d261c8aba7034fdd9ed18dbd66d4 not found: ID does not exist" Jul 13 21:06:13.767154 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:13.767131 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:14.833308 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:14.833221 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8lq85\" is forbidden: User \"system:node:ip-10-0-130-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-205.ec2.internal' and this object" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8lq85" Jul 13 21:06:14.835428 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:14.835401 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da01438-c653-4fae-b4a8-a6cff51ca7d8" path="/var/lib/kubelet/pods/5da01438-c653-4fae-b4a8-a6cff51ca7d8/volumes" Jul 13 21:06:23.758591 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:23.758550 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:23.759027 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:23.758637 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:23.759027 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:23.758680 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zjvfb" Jul 13 21:06:23.880364 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:23.880329 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:23.880580 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:23.880557 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" podUID="f4ea7624-43ec-410b-80aa-77e4423d3883" containerName="manager" containerID="cri-o://9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83" gracePeriod=10 Jul 13 21:06:24.146821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.146799 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:24.216337 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.216312 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt\") pod \"f4ea7624-43ec-410b-80aa-77e4423d3883\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " Jul 13 21:06:24.216469 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.216403 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume\") pod \"f4ea7624-43ec-410b-80aa-77e4423d3883\" (UID: \"f4ea7624-43ec-410b-80aa-77e4423d3883\") " Jul 13 21:06:24.216803 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.216783 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f4ea7624-43ec-410b-80aa-77e4423d3883" (UID: "f4ea7624-43ec-410b-80aa-77e4423d3883"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:06:24.218424 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.218406 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt" (OuterVolumeSpecName: "kube-api-access-fzgtt") pod "f4ea7624-43ec-410b-80aa-77e4423d3883" (UID: "f4ea7624-43ec-410b-80aa-77e4423d3883"). InnerVolumeSpecName "kube-api-access-fzgtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:06:24.282859 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.282831 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:06:24.283196 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.283184 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ea7624-43ec-410b-80aa-77e4423d3883" containerName="manager" Jul 13 21:06:24.283239 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.283198 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ea7624-43ec-410b-80aa-77e4423d3883" containerName="manager" Jul 13 21:06:24.283276 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.283256 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ea7624-43ec-410b-80aa-77e4423d3883" containerName="manager" Jul 13 21:06:24.286366 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.286350 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.303063 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.302995 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:06:24.317852 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.317831 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgtt\" (UniqueName: \"kubernetes.io/projected/f4ea7624-43ec-410b-80aa-77e4423d3883-kube-api-access-fzgtt\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:24.317852 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.317852 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4ea7624-43ec-410b-80aa-77e4423d3883-extensions-socket-volume\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:24.419150 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.419119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.419273 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.419164 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.520543 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.520511 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.520690 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.520556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.520987 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.520968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.533276 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.533252 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b\") pod \"kuadrant-operator-controller-manager-86878c769f-fskml\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.596630 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.596573 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.730946 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.730915 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:06:24.733505 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:24.733477 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415778b1_ffdd_4815_bb3f_6690d89ab180.slice/crio-75693a763ad476ab4feb2d47804d94dac95404b23f4fea0936506815ff6bca0b WatchSource:0}: Error finding container 75693a763ad476ab4feb2d47804d94dac95404b23f4fea0936506815ff6bca0b: Status 404 returned error can't find the container with id 75693a763ad476ab4feb2d47804d94dac95404b23f4fea0936506815ff6bca0b Jul 13 21:06:24.819893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.819865 2561 generic.go:356] "Generic (PLEG): container finished" podID="f4ea7624-43ec-410b-80aa-77e4423d3883" containerID="9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83" exitCode=0 Jul 13 21:06:24.820220 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.819916 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" Jul 13 21:06:24.820220 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.819945 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" event={"ID":"f4ea7624-43ec-410b-80aa-77e4423d3883","Type":"ContainerDied","Data":"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83"} Jul 13 21:06:24.820220 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.819989 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl" event={"ID":"f4ea7624-43ec-410b-80aa-77e4423d3883","Type":"ContainerDied","Data":"1a1eccdcc7707e83f4061a6ed4d1b8b1383cffd784a0c9547b17d11e3adc98c1"} Jul 13 21:06:24.820220 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.820018 2561 scope.go:117] "RemoveContainer" containerID="9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83" Jul 13 21:06:24.821694 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.821668 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" event={"ID":"415778b1-ffdd-4815-bb3f-6690d89ab180","Type":"ContainerStarted","Data":"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3"} Jul 13 21:06:24.821800 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.821709 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" event={"ID":"415778b1-ffdd-4815-bb3f-6690d89ab180","Type":"ContainerStarted","Data":"75693a763ad476ab4feb2d47804d94dac95404b23f4fea0936506815ff6bca0b"} Jul 13 21:06:24.821877 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.821800 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:24.828345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.828328 2561 scope.go:117] "RemoveContainer" containerID="9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83" Jul 13 21:06:24.828607 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:24.828590 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83\": container with ID starting with 9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83 not found: ID does not exist" containerID="9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83" Jul 13 21:06:24.828691 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.828615 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83"} err="failed to get container status \"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83\": rpc error: code = NotFound desc = could not find container \"9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83\": container with ID starting with 9cd40e73a4400f6ffbfd44f67ef8c9960d4acf96fe3608f65e4b38b0fbf07b83 not found: ID does not exist" Jul 13 21:06:24.851012 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.850961 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" podStartSLOduration=0.850942781 podStartE2EDuration="850.942781ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:24.848191644 +0000 UTC m=+760.577251545" watchObservedRunningTime="2026-07-13 21:06:24.850942781 +0000 UTC m=+760.580002683" Jul 13 21:06:24.874229 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.874203 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:24.880819 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:24.880796 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5hzcl"] Jul 13 21:06:26.834743 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:26.834710 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ea7624-43ec-410b-80aa-77e4423d3883" path="/var/lib/kubelet/pods/f4ea7624-43ec-410b-80aa-77e4423d3883/volumes" Jul 13 21:06:35.830140 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:35.830103 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:06:35.872892 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:35.872860 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:35.873075 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:35.873034 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" podUID="53eedecb-9b6b-4866-9e40-35098846447e" containerName="manager" containerID="cri-o://144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528" gracePeriod=10 Jul 13 21:06:36.143259 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.143231 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:36.315948 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.315919 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume\") pod \"53eedecb-9b6b-4866-9e40-35098846447e\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " Jul 13 21:06:36.316104 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.315964 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtslx\" (UniqueName: \"kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx\") pod \"53eedecb-9b6b-4866-9e40-35098846447e\" (UID: \"53eedecb-9b6b-4866-9e40-35098846447e\") " Jul 13 21:06:36.316295 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.316262 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53eedecb-9b6b-4866-9e40-35098846447e" (UID: "53eedecb-9b6b-4866-9e40-35098846447e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:06:36.318077 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.318052 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx" (OuterVolumeSpecName: "kube-api-access-rtslx") pod "53eedecb-9b6b-4866-9e40-35098846447e" (UID: "53eedecb-9b6b-4866-9e40-35098846447e"). InnerVolumeSpecName "kube-api-access-rtslx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:06:36.417161 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.417082 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtslx\" (UniqueName: \"kubernetes.io/projected/53eedecb-9b6b-4866-9e40-35098846447e-kube-api-access-rtslx\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:36.417161 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.417108 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53eedecb-9b6b-4866-9e40-35098846447e-extensions-socket-volume\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:06:36.880143 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.880107 2561 generic.go:356] "Generic (PLEG): container finished" podID="53eedecb-9b6b-4866-9e40-35098846447e" containerID="144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528" exitCode=0 Jul 13 21:06:36.880541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.880168 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" event={"ID":"53eedecb-9b6b-4866-9e40-35098846447e","Type":"ContainerDied","Data":"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528"} Jul 13 21:06:36.880541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.880174 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" Jul 13 21:06:36.880541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.880208 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf" event={"ID":"53eedecb-9b6b-4866-9e40-35098846447e","Type":"ContainerDied","Data":"26f07217f97555e455c4ef10f0e40335f0037816f5d64bce06af73e8b4e61d4a"} Jul 13 21:06:36.880541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.880232 2561 scope.go:117] "RemoveContainer" containerID="144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528" Jul 13 21:06:36.888901 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.888881 2561 scope.go:117] "RemoveContainer" containerID="144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528" Jul 13 21:06:36.889159 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:06:36.889138 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528\": container with ID starting with 144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528 not found: ID does not exist" containerID="144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528" Jul 13 21:06:36.889232 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.889173 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528"} err="failed to get container status \"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528\": rpc error: code = NotFound desc = could not find container \"144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528\": container with ID starting with 144b328864cf700c0ca2027189e816c4fc47ba6b34241bf42d2cc356d5cc1528 not found: ID does not exist" Jul 13 21:06:36.902326 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.902304 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:36.908876 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:36.908845 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l57pf"] Jul 13 21:06:38.834407 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:38.834374 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53eedecb-9b6b-4866-9e40-35098846447e" path="/var/lib/kubelet/pods/53eedecb-9b6b-4866-9e40-35098846447e/volumes" Jul 13 21:06:40.307626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.307584 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw"] Jul 13 21:06:40.308180 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.308156 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53eedecb-9b6b-4866-9e40-35098846447e" containerName="manager" Jul 13 21:06:40.308180 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.308177 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53eedecb-9b6b-4866-9e40-35098846447e" containerName="manager" Jul 13 21:06:40.308296 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.308267 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53eedecb-9b6b-4866-9e40-35098846447e" containerName="manager" Jul 13 21:06:40.311287 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.311267 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.314041 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.314017 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gt2mw\"" Jul 13 21:06:40.314130 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.314060 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 21:06:40.324562 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.324542 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw"] Jul 13 21:06:40.454252 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454220 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454279 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454298 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454316 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhsj\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-kube-api-access-xbhsj\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454372 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454413 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.454798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.454522 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555206 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555242 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555257 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555378 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555423 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555440 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555525 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555567 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhsj\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-kube-api-access-xbhsj\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.555861 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.556019 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555999 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.556072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.555998 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.556072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.556051 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.556365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.556347 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.557891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.557838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.557984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.557885 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.563417 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.563389 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.563977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.563953 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhsj\" (UniqueName: \"kubernetes.io/projected/71ac7e7c-12a7-4050-84a0-023a08a6f1a2-kube-api-access-xbhsj\") pod \"maas-default-gateway-openshift-default-56654dcf7-q8vsw\" (UID: \"71ac7e7c-12a7-4050-84a0-023a08a6f1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.620854 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.620824 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:40.752958 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.752933 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw"] Jul 13 21:06:40.755054 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:40.755026 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ac7e7c_12a7_4050_84a0_023a08a6f1a2.slice/crio-56d19c2fb79ad23dc52a2efe0c00afdfe3b33b3a075b6b34662a25463378a1fa WatchSource:0}: Error finding container 56d19c2fb79ad23dc52a2efe0c00afdfe3b33b3a075b6b34662a25463378a1fa: Status 404 returned error can't find the container with id 56d19c2fb79ad23dc52a2efe0c00afdfe3b33b3a075b6b34662a25463378a1fa Jul 13 21:06:40.903023 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:40.902985 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" event={"ID":"71ac7e7c-12a7-4050-84a0-023a08a6f1a2","Type":"ContainerStarted","Data":"56d19c2fb79ad23dc52a2efe0c00afdfe3b33b3a075b6b34662a25463378a1fa"} Jul 13 21:06:43.800265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:43.800232 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:06:43.800508 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:43.800302 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:06:43.800508 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:43.800331 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:06:43.922231 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:43.922196 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" event={"ID":"71ac7e7c-12a7-4050-84a0-023a08a6f1a2","Type":"ContainerStarted","Data":"e4c0d47a81ddf5ca3fd4efb5bce9bf6b070968aab3c9ad17464ecac521be1bb6"} Jul 13 21:06:43.943207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:43.943148 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" podStartSLOduration=0.900182031 podStartE2EDuration="3.943133911s" totalImagesPullingTime="3.04295188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:43.941877112 +0000 UTC m=+779.670937013" watchObservedRunningTime="2026-07-13 21:06:43.943133911 +0000 UTC m=+779.672193807" Jul 13 21:06:44.622007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:44.621974 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:44.626419 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:44.626395 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:44.929827 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:44.929154 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:44.932542 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:44.932517 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:06:52.005677 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.005641 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:06:52.010995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.010973 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.013775 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.013736 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 21:06:52.013878 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.013736 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h6kj7\"" Jul 13 21:06:52.018725 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.018595 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:06:52.055226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.055180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z976s\" (UniqueName: \"kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.055411 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.055285 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.075786 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.075738 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:06:52.156425 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.156383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z976s\" (UniqueName: \"kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.156601 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.156477 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.157264 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.157238 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.164296 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.164263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z976s\" (UniqueName: \"kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s\") pod \"limitador-limitador-64c94fd64f-2r26q\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.322908 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.322824 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:52.456533 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.456508 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:06:52.458900 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:52.458873 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877b0dfe_271a_4fc8_a317_a0a124765de3.slice/crio-e6812d387f6050f337920970770be5d363a1f7847dbee8a1cfe1960401cdc905 WatchSource:0}: Error finding container e6812d387f6050f337920970770be5d363a1f7847dbee8a1cfe1960401cdc905: Status 404 returned error can't find the container with id e6812d387f6050f337920970770be5d363a1f7847dbee8a1cfe1960401cdc905 Jul 13 21:06:52.914569 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.914535 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:06:52.919107 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.919088 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:06:52.921349 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.921326 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7msf9\"" Jul 13 21:06:52.922642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.922619 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:06:52.962361 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.962329 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq8mw\" (UniqueName: \"kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw\") pod \"authorino-75d556bd77-xw2ql\" (UID: \"2d482ad0-3dc0-4052-b87e-6d6fbb6752df\") " pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:06:52.969773 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:52.969737 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" event={"ID":"877b0dfe-271a-4fc8-a317-a0a124765de3","Type":"ContainerStarted","Data":"e6812d387f6050f337920970770be5d363a1f7847dbee8a1cfe1960401cdc905"} Jul 13 21:06:53.063671 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:53.063643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq8mw\" (UniqueName: \"kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw\") pod \"authorino-75d556bd77-xw2ql\" (UID: \"2d482ad0-3dc0-4052-b87e-6d6fbb6752df\") " pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:06:53.074454 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:53.074427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq8mw\" (UniqueName: \"kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw\") pod \"authorino-75d556bd77-xw2ql\" (UID: \"2d482ad0-3dc0-4052-b87e-6d6fbb6752df\") " pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:06:53.230391 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:53.230311 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:06:53.396639 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:53.396579 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:06:53.398851 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:06:53.398806 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d482ad0_3dc0_4052_b87e_6d6fbb6752df.slice/crio-b96743c1ad26af4dd8e60988a41cb400e1918bab96de317e3023ae28086118d8 WatchSource:0}: Error finding container b96743c1ad26af4dd8e60988a41cb400e1918bab96de317e3023ae28086118d8: Status 404 returned error can't find the container with id b96743c1ad26af4dd8e60988a41cb400e1918bab96de317e3023ae28086118d8 Jul 13 21:06:53.978194 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:53.978153 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xw2ql" event={"ID":"2d482ad0-3dc0-4052-b87e-6d6fbb6752df","Type":"ContainerStarted","Data":"b96743c1ad26af4dd8e60988a41cb400e1918bab96de317e3023ae28086118d8"} Jul 13 21:06:55.991679 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:55.991631 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" event={"ID":"877b0dfe-271a-4fc8-a317-a0a124765de3","Type":"ContainerStarted","Data":"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377"} Jul 13 21:06:55.992167 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:55.991930 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:06:56.009573 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:56.009515 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" podStartSLOduration=1.958257309 podStartE2EDuration="5.009501095s" totalImagesPullingTime="3.051243786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:56.007442934 +0000 UTC m=+791.736502839" watchObservedRunningTime="2026-07-13 21:06:56.009501095 +0000 UTC m=+791.738560995" Jul 13 21:06:58.004398 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:58.004357 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xw2ql" event={"ID":"2d482ad0-3dc0-4052-b87e-6d6fbb6752df","Type":"ContainerStarted","Data":"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1"} Jul 13 21:06:58.017905 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:06:58.017863 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-xw2ql" podStartSLOduration=1.503211935 podStartE2EDuration="6.017845275s" totalImagesPullingTime="4.51463334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:06:58.017150808 +0000 UTC m=+793.746210729" watchObservedRunningTime="2026-07-13 21:06:58.017845275 +0000 UTC m=+793.746905174" Jul 13 21:07:06.997343 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:06.997306 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:07:08.969383 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:08.969349 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:07:08.969825 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:08.969561 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" podUID="877b0dfe-271a-4fc8-a317-a0a124765de3" containerName="limitador" containerID="cri-o://2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377" gracePeriod=30 Jul 13 21:07:09.524188 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.524163 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:07:09.604892 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.604868 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z976s\" (UniqueName: \"kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s\") pod \"877b0dfe-271a-4fc8-a317-a0a124765de3\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " Jul 13 21:07:09.605030 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.604932 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file\") pod \"877b0dfe-271a-4fc8-a317-a0a124765de3\" (UID: \"877b0dfe-271a-4fc8-a317-a0a124765de3\") " Jul 13 21:07:09.605228 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.605207 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file" (OuterVolumeSpecName: "config-file") pod "877b0dfe-271a-4fc8-a317-a0a124765de3" (UID: "877b0dfe-271a-4fc8-a317-a0a124765de3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:07:09.606931 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.606903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s" (OuterVolumeSpecName: "kube-api-access-z976s") pod "877b0dfe-271a-4fc8-a317-a0a124765de3" (UID: "877b0dfe-271a-4fc8-a317-a0a124765de3"). InnerVolumeSpecName "kube-api-access-z976s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:07:09.705906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.705877 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/877b0dfe-271a-4fc8-a317-a0a124765de3-config-file\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:09.705906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:09.705906 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z976s\" (UniqueName: \"kubernetes.io/projected/877b0dfe-271a-4fc8-a317-a0a124765de3-kube-api-access-z976s\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:10.062560 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.062481 2561 generic.go:356] "Generic (PLEG): container finished" podID="877b0dfe-271a-4fc8-a317-a0a124765de3" containerID="2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377" exitCode=0 Jul 13 21:07:10.062560 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.062526 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" Jul 13 21:07:10.063021 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.062567 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" event={"ID":"877b0dfe-271a-4fc8-a317-a0a124765de3","Type":"ContainerDied","Data":"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377"} Jul 13 21:07:10.063021 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.062625 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2r26q" event={"ID":"877b0dfe-271a-4fc8-a317-a0a124765de3","Type":"ContainerDied","Data":"e6812d387f6050f337920970770be5d363a1f7847dbee8a1cfe1960401cdc905"} Jul 13 21:07:10.063021 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.062648 2561 scope.go:117] "RemoveContainer" containerID="2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377" Jul 13 21:07:10.071642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.071619 2561 scope.go:117] "RemoveContainer" containerID="2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377" Jul 13 21:07:10.071895 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:07:10.071878 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377\": container with ID starting with 2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377 not found: ID does not exist" containerID="2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377" Jul 13 21:07:10.071966 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.071903 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377"} err="failed to get container status \"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377\": rpc error: code = NotFound desc = could not find container \"2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377\": container with ID starting with 2ab1fe8cc726819a0c81bb5a63522834d7869782ac9e606370f1d14f80ba0377 not found: ID does not exist" Jul 13 21:07:10.083193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.083168 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:07:10.086911 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.086890 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2r26q"] Jul 13 21:07:10.834096 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:10.834064 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877b0dfe-271a-4fc8-a317-a0a124765de3" path="/var/lib/kubelet/pods/877b0dfe-271a-4fc8-a317-a0a124765de3/volumes" Jul 13 21:07:11.765294 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.765257 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg"] Jul 13 21:07:11.765770 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.765658 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877b0dfe-271a-4fc8-a317-a0a124765de3" containerName="limitador" Jul 13 21:07:11.765770 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.765671 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="877b0dfe-271a-4fc8-a317-a0a124765de3" containerName="limitador" Jul 13 21:07:11.765913 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.765785 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="877b0dfe-271a-4fc8-a317-a0a124765de3" containerName="limitador" Jul 13 21:07:11.769037 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.769015 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.771506 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.771475 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 13 21:07:11.771641 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.771618 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 21:07:11.771701 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.771619 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-db9lc\"" Jul 13 21:07:11.772409 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.772394 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 21:07:11.778688 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.778667 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg"] Jul 13 21:07:11.826297 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.826260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-data\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.826462 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.826401 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb97t\" (UniqueName: \"kubernetes.io/projected/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-kube-api-access-vb97t\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.927705 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.927670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb97t\" (UniqueName: \"kubernetes.io/projected/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-kube-api-access-vb97t\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.927891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.927724 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-data\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.928154 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.928137 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-data\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:11.935707 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:11.935680 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb97t\" (UniqueName: \"kubernetes.io/projected/801ece9b-849a-4dfd-acaf-5a4c39cd2b01-kube-api-access-vb97t\") pod \"postgres-69fb4594cf-t8pjg\" (UID: \"801ece9b-849a-4dfd-acaf-5a4c39cd2b01\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:12.080135 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:12.080054 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:12.213660 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:12.213635 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg"] Jul 13 21:07:12.215992 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:07:12.215959 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801ece9b_849a_4dfd_acaf_5a4c39cd2b01.slice/crio-681ae2b6016b55abb135ec9b0d04e9da731731236c949a87eeaa6656be999403 WatchSource:0}: Error finding container 681ae2b6016b55abb135ec9b0d04e9da731731236c949a87eeaa6656be999403: Status 404 returned error can't find the container with id 681ae2b6016b55abb135ec9b0d04e9da731731236c949a87eeaa6656be999403 Jul 13 21:07:13.078671 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:13.078628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" event={"ID":"801ece9b-849a-4dfd-acaf-5a4c39cd2b01","Type":"ContainerStarted","Data":"681ae2b6016b55abb135ec9b0d04e9da731731236c949a87eeaa6656be999403"} Jul 13 21:07:18.106450 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:18.106413 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" event={"ID":"801ece9b-849a-4dfd-acaf-5a4c39cd2b01","Type":"ContainerStarted","Data":"d7c6b1ec3365ba52050358f2d498cc5606c1ceac199113d128ad00540c82bbe9"} Jul 13 21:07:18.106879 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:18.106496 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:18.124212 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:18.124148 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" podStartSLOduration=1.778604921 podStartE2EDuration="7.124126828s" totalImagesPullingTime="5.345521907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:07:18.123323093 +0000 UTC m=+813.852382994" watchObservedRunningTime="2026-07-13 21:07:18.124126828 +0000 UTC m=+813.853186729" Jul 13 21:07:24.142299 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.142274 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-t8pjg" Jul 13 21:07:24.697076 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.697041 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:24.700690 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.700669 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:24.706993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.706956 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:24.845460 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.845430 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lr8\" (UniqueName: \"kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8\") pod \"authorino-5d4b67c446-qzdbx\" (UID: \"f2d3da4e-6b27-4928-b8d1-dc2a806326be\") " pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:24.946375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.946337 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lr8\" (UniqueName: \"kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8\") pod \"authorino-5d4b67c446-qzdbx\" (UID: \"f2d3da4e-6b27-4928-b8d1-dc2a806326be\") " pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:24.957216 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.957153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lr8\" (UniqueName: \"kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8\") pod \"authorino-5d4b67c446-qzdbx\" (UID: \"f2d3da4e-6b27-4928-b8d1-dc2a806326be\") " pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:24.961713 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.961691 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:24.961950 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.961935 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:24.988016 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.987992 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:07:24.992294 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.992275 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:24.994766 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.994737 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 13 21:07:24.997949 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:24.997930 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:07:25.103534 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.103509 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:25.105168 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:07:25.105139 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2d3da4e_6b27_4928_b8d1_dc2a806326be.slice/crio-5f9812a24766d4bde1ca6a78caba92c3e462954d4c06b1732d5c39aefdc2de64 WatchSource:0}: Error finding container 5f9812a24766d4bde1ca6a78caba92c3e462954d4c06b1732d5c39aefdc2de64: Status 404 returned error can't find the container with id 5f9812a24766d4bde1ca6a78caba92c3e462954d4c06b1732d5c39aefdc2de64 Jul 13 21:07:25.141213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.141183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" event={"ID":"f2d3da4e-6b27-4928-b8d1-dc2a806326be","Type":"ContainerStarted","Data":"5f9812a24766d4bde1ca6a78caba92c3e462954d4c06b1732d5c39aefdc2de64"} Jul 13 21:07:25.147598 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.147576 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmvqw\" (UniqueName: \"kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.147903 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.147683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.248395 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.248319 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmvqw\" (UniqueName: \"kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.248525 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.248443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.250824 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.250801 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.255831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.255808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmvqw\" (UniqueName: \"kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw\") pod \"authorino-775797f4d4-wcb72\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.304607 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.304580 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:07:25.437946 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:25.437916 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:07:25.441294 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:07:25.441268 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc627e29_9cbd_43b4_8736_89ae75456106.slice/crio-e5a4cf89880a163e7f7946d6ce6c7c7762d92f4fb971987f25ae76c35a7bb24b WatchSource:0}: Error finding container e5a4cf89880a163e7f7946d6ce6c7c7762d92f4fb971987f25ae76c35a7bb24b: Status 404 returned error can't find the container with id e5a4cf89880a163e7f7946d6ce6c7c7762d92f4fb971987f25ae76c35a7bb24b Jul 13 21:07:26.147459 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.147420 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" event={"ID":"f2d3da4e-6b27-4928-b8d1-dc2a806326be","Type":"ContainerStarted","Data":"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b"} Jul 13 21:07:26.147658 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.147434 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" podUID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" containerName="authorino" containerID="cri-o://ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b" gracePeriod=30 Jul 13 21:07:26.149187 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.149163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775797f4d4-wcb72" event={"ID":"cc627e29-9cbd-43b4-8736-89ae75456106","Type":"ContainerStarted","Data":"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516"} Jul 13 21:07:26.149283 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.149197 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775797f4d4-wcb72" event={"ID":"cc627e29-9cbd-43b4-8736-89ae75456106","Type":"ContainerStarted","Data":"e5a4cf89880a163e7f7946d6ce6c7c7762d92f4fb971987f25ae76c35a7bb24b"} Jul 13 21:07:26.162530 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.162484 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" podStartSLOduration=1.718206275 podStartE2EDuration="2.162470975s" totalImagesPullingTime="444.2647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:07:26.161185712 +0000 UTC m=+821.890245637" watchObservedRunningTime="2026-07-13 21:07:26.162470975 +0000 UTC m=+821.891530874" Jul 13 21:07:26.176242 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.176200 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-775797f4d4-wcb72" podStartSLOduration=1.803237927 podStartE2EDuration="2.176187059s" totalImagesPullingTime="372.949132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:07:26.175252211 +0000 UTC m=+821.904312116" watchObservedRunningTime="2026-07-13 21:07:26.176187059 +0000 UTC m=+821.905247005" Jul 13 21:07:26.203074 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.203044 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:07:26.203259 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.203232 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-xw2ql" podUID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" containerName="authorino" containerID="cri-o://d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1" gracePeriod=30 Jul 13 21:07:26.480062 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.480038 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:26.482915 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.482894 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:07:26.660108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.660038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59lr8\" (UniqueName: \"kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8\") pod \"f2d3da4e-6b27-4928-b8d1-dc2a806326be\" (UID: \"f2d3da4e-6b27-4928-b8d1-dc2a806326be\") " Jul 13 21:07:26.660108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.660077 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq8mw\" (UniqueName: \"kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw\") pod \"2d482ad0-3dc0-4052-b87e-6d6fbb6752df\" (UID: \"2d482ad0-3dc0-4052-b87e-6d6fbb6752df\") " Jul 13 21:07:26.662089 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.662062 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw" (OuterVolumeSpecName: "kube-api-access-gq8mw") pod "2d482ad0-3dc0-4052-b87e-6d6fbb6752df" (UID: "2d482ad0-3dc0-4052-b87e-6d6fbb6752df"). InnerVolumeSpecName "kube-api-access-gq8mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:07:26.662213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.662110 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8" (OuterVolumeSpecName: "kube-api-access-59lr8") pod "f2d3da4e-6b27-4928-b8d1-dc2a806326be" (UID: "f2d3da4e-6b27-4928-b8d1-dc2a806326be"). InnerVolumeSpecName "kube-api-access-59lr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:07:26.761015 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.760987 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59lr8\" (UniqueName: \"kubernetes.io/projected/f2d3da4e-6b27-4928-b8d1-dc2a806326be-kube-api-access-59lr8\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:26.761015 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:26.761013 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq8mw\" (UniqueName: \"kubernetes.io/projected/2d482ad0-3dc0-4052-b87e-6d6fbb6752df-kube-api-access-gq8mw\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:27.155360 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.155323 2561 generic.go:356] "Generic (PLEG): container finished" podID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" containerID="d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1" exitCode=0 Jul 13 21:07:27.155845 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.155375 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xw2ql" Jul 13 21:07:27.155845 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.155406 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xw2ql" event={"ID":"2d482ad0-3dc0-4052-b87e-6d6fbb6752df","Type":"ContainerDied","Data":"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1"} Jul 13 21:07:27.155845 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.155446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xw2ql" event={"ID":"2d482ad0-3dc0-4052-b87e-6d6fbb6752df","Type":"ContainerDied","Data":"b96743c1ad26af4dd8e60988a41cb400e1918bab96de317e3023ae28086118d8"} Jul 13 21:07:27.155845 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.155459 2561 scope.go:117] "RemoveContainer" containerID="d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1" Jul 13 21:07:27.156920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.156678 2561 generic.go:356] "Generic (PLEG): container finished" podID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" containerID="ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b" exitCode=0 Jul 13 21:07:27.156920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.156712 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" Jul 13 21:07:27.156920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.156711 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" event={"ID":"f2d3da4e-6b27-4928-b8d1-dc2a806326be","Type":"ContainerDied","Data":"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b"} Jul 13 21:07:27.156920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.156785 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-qzdbx" event={"ID":"f2d3da4e-6b27-4928-b8d1-dc2a806326be","Type":"ContainerDied","Data":"5f9812a24766d4bde1ca6a78caba92c3e462954d4c06b1732d5c39aefdc2de64"} Jul 13 21:07:27.164230 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.164212 2561 scope.go:117] "RemoveContainer" containerID="d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1" Jul 13 21:07:27.164490 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:07:27.164473 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1\": container with ID starting with d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1 not found: ID does not exist" containerID="d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1" Jul 13 21:07:27.164548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.164498 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1"} err="failed to get container status \"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1\": rpc error: code = NotFound desc = could not find container \"d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1\": container with ID starting with d6a077671d2b52cc5dea1060e654882a67f5244b576cb7169014a40623ef3cb1 not found: ID does not exist" Jul 13 21:07:27.164548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.164516 2561 scope.go:117] "RemoveContainer" containerID="ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b" Jul 13 21:07:27.172578 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.172547 2561 scope.go:117] "RemoveContainer" containerID="ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b" Jul 13 21:07:27.172922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.172877 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:07:27.172922 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:07:27.172884 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b\": container with ID starting with ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b not found: ID does not exist" containerID="ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b" Jul 13 21:07:27.173116 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.172916 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b"} err="failed to get container status \"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b\": rpc error: code = NotFound desc = could not find container \"ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b\": container with ID starting with ec49e2e4decab3ab7e259c722d77a7a9ef10de9608df09618812d6d50510999b not found: ID does not exist" Jul 13 21:07:27.174726 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.174707 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xw2ql"] Jul 13 21:07:27.184301 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.184280 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:27.188489 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:27.188469 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-qzdbx"] Jul 13 21:07:28.834566 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:28.834521 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" path="/var/lib/kubelet/pods/2d482ad0-3dc0-4052-b87e-6d6fbb6752df/volumes" Jul 13 21:07:28.835173 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:28.834992 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" path="/var/lib/kubelet/pods/f2d3da4e-6b27-4928-b8d1-dc2a806326be/volumes" Jul 13 21:07:30.939502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939469 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:30.939927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939871 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" containerName="authorino" Jul 13 21:07:30.939927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939885 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" containerName="authorino" Jul 13 21:07:30.939927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939898 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" containerName="authorino" Jul 13 21:07:30.939927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939904 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" containerName="authorino" Jul 13 21:07:30.940081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939964 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d482ad0-3dc0-4052-b87e-6d6fbb6752df" containerName="authorino" Jul 13 21:07:30.940081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.939971 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d3da4e-6b27-4928-b8d1-dc2a806326be" containerName="authorino" Jul 13 21:07:30.946116 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.946096 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:30.948542 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.948523 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 21:07:30.948810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.948784 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k4bbc\"" Jul 13 21:07:30.949372 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.949357 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 21:07:30.955157 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:30.955135 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:31.099529 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.099493 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.099709 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.099551 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8jh\" (UniqueName: \"kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.200790 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.200683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.200790 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.200784 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8jh\" (UniqueName: \"kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.203116 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.203093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.208971 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.208940 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8jh\" (UniqueName: \"kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh\") pod \"maas-controller-86556c6b6-vbspx\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.257743 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.257711 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:31.398675 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:31.398649 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:31.400916 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:07:31.400889 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b04514_e721_4d61_b9c3_8f790e4d17ee.slice/crio-b614137b59ab87b58d9495a91e5313e11ef8c6eb7c96a7f39d234de93272be8b WatchSource:0}: Error finding container b614137b59ab87b58d9495a91e5313e11ef8c6eb7c96a7f39d234de93272be8b: Status 404 returned error can't find the container with id b614137b59ab87b58d9495a91e5313e11ef8c6eb7c96a7f39d234de93272be8b Jul 13 21:07:32.191766 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:32.191672 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86556c6b6-vbspx" event={"ID":"65b04514-e721-4d61-b9c3-8f790e4d17ee","Type":"ContainerStarted","Data":"b614137b59ab87b58d9495a91e5313e11ef8c6eb7c96a7f39d234de93272be8b"} Jul 13 21:07:34.203815 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:34.203777 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86556c6b6-vbspx" event={"ID":"65b04514-e721-4d61-b9c3-8f790e4d17ee","Type":"ContainerStarted","Data":"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee"} Jul 13 21:07:34.204207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:34.203896 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:34.231806 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:34.231727 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86556c6b6-vbspx" podStartSLOduration=2.02202788 podStartE2EDuration="4.231712324s" totalImagesPullingTime="2.209684444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:07:34.229386115 +0000 UTC m=+829.958446014" watchObservedRunningTime="2026-07-13 21:07:34.231712324 +0000 UTC m=+829.960772224" Jul 13 21:07:45.210544 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.210510 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:45.524332 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.524247 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:45.524520 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.524472 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86556c6b6-vbspx" podUID="65b04514-e721-4d61-b9c3-8f790e4d17ee" containerName="manager" containerID="cri-o://383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee" gracePeriod=30 Jul 13 21:07:45.773983 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.773958 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:45.822622 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.822558 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert\") pod \"65b04514-e721-4d61-b9c3-8f790e4d17ee\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " Jul 13 21:07:45.822622 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.822591 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps8jh\" (UniqueName: \"kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh\") pod \"65b04514-e721-4d61-b9c3-8f790e4d17ee\" (UID: \"65b04514-e721-4d61-b9c3-8f790e4d17ee\") " Jul 13 21:07:45.824852 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.824824 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "65b04514-e721-4d61-b9c3-8f790e4d17ee" (UID: "65b04514-e721-4d61-b9c3-8f790e4d17ee"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:07:45.824964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.824936 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh" (OuterVolumeSpecName: "kube-api-access-ps8jh") pod "65b04514-e721-4d61-b9c3-8f790e4d17ee" (UID: "65b04514-e721-4d61-b9c3-8f790e4d17ee"). InnerVolumeSpecName "kube-api-access-ps8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:07:45.924076 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.924046 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b04514-e721-4d61-b9c3-8f790e4d17ee-webhook-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:45.924076 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:45.924071 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8jh\" (UniqueName: \"kubernetes.io/projected/65b04514-e721-4d61-b9c3-8f790e4d17ee-kube-api-access-ps8jh\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:07:46.262558 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.262525 2561 generic.go:356] "Generic (PLEG): container finished" podID="65b04514-e721-4d61-b9c3-8f790e4d17ee" containerID="383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee" exitCode=0 Jul 13 21:07:46.263018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.262583 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86556c6b6-vbspx" Jul 13 21:07:46.263018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.262615 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86556c6b6-vbspx" event={"ID":"65b04514-e721-4d61-b9c3-8f790e4d17ee","Type":"ContainerDied","Data":"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee"} Jul 13 21:07:46.263018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.262666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86556c6b6-vbspx" event={"ID":"65b04514-e721-4d61-b9c3-8f790e4d17ee","Type":"ContainerDied","Data":"b614137b59ab87b58d9495a91e5313e11ef8c6eb7c96a7f39d234de93272be8b"} Jul 13 21:07:46.263018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.262687 2561 scope.go:117] "RemoveContainer" containerID="383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee" Jul 13 21:07:46.272057 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.272040 2561 scope.go:117] "RemoveContainer" containerID="383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee" Jul 13 21:07:46.272289 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:07:46.272269 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee\": container with ID starting with 383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee not found: ID does not exist" containerID="383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee" Jul 13 21:07:46.272335 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.272296 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee"} err="failed to get container status \"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee\": rpc error: code = NotFound desc = could not find container \"383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee\": container with ID starting with 383b317264c251e8ca85e258cdc4070632e299f1be01a53d4d71465d469313ee not found: ID does not exist" Jul 13 21:07:46.285640 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.285615 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:46.289113 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.289088 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86556c6b6-vbspx"] Jul 13 21:07:46.835414 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:07:46.835377 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b04514-e721-4d61-b9c3-8f790e4d17ee" path="/var/lib/kubelet/pods/65b04514-e721-4d61-b9c3-8f790e4d17ee/volumes" Jul 13 21:08:05.301682 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.301647 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff844954-j5st8"] Jul 13 21:08:05.302151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.302025 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b04514-e721-4d61-b9c3-8f790e4d17ee" containerName="manager" Jul 13 21:08:05.302151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.302036 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b04514-e721-4d61-b9c3-8f790e4d17ee" containerName="manager" Jul 13 21:08:05.302151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.302097 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b04514-e721-4d61-b9c3-8f790e4d17ee" containerName="manager" Jul 13 21:08:05.317023 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.316993 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.320286 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.320259 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff844954-j5st8"] Jul 13 21:08:05.320990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.320962 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 21:08:05.321098 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.320969 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-j4tgz\"" Jul 13 21:08:05.386238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.386206 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lmb\" (UniqueName: \"kubernetes.io/projected/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-kube-api-access-65lmb\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.386238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.386242 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-openshift-service-ca\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.386448 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.386263 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.486968 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.486933 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.487117 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.487062 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lmb\" (UniqueName: \"kubernetes.io/projected/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-kube-api-access-65lmb\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.487117 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:08:05.487091 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 13 21:08:05.487258 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:08:05.487179 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls podName:d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d nodeName:}" failed. No retries permitted until 2026-07-13 21:08:05.987157584 +0000 UTC m=+861.716217480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls") pod "maas-api-6ff844954-j5st8" (UID: "d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d") : secret "maas-api-serving-cert" not found Jul 13 21:08:05.487258 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.487092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-openshift-service-ca\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.487702 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.487685 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-openshift-service-ca\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.497059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.497034 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lmb\" (UniqueName: \"kubernetes.io/projected/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-kube-api-access-65lmb\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.831319 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.831284 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-zjxzg"] Jul 13 21:08:05.834120 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.834103 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:05.837849 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.837822 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 21:08:05.845515 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.845496 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-zjxzg"] Jul 13 21:08:05.890202 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.890130 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-pswft"] Jul 13 21:08:05.890433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.890405 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:05.890548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.890460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvbb\" (UniqueName: \"kubernetes.io/projected/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-kube-api-access-jsvbb\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:05.894595 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.894572 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:05.897259 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.897238 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bcqcs\"" Jul 13 21:08:05.908482 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.908458 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-pswft"] Jul 13 21:08:05.991511 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.991474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbvm\" (UniqueName: \"kubernetes.io/projected/85dc4ad8-de67-47d1-bd81-06a1998c80a6-kube-api-access-tnbvm\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:05.991661 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.991571 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:05.991661 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.991597 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dc4ad8-de67-47d1-bd81-06a1998c80a6-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:05.991661 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.991651 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:05.991864 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.991683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvbb\" (UniqueName: \"kubernetes.io/projected/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-kube-api-access-jsvbb\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:05.991864 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:08:05.991695 2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 13 21:08:05.991864 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:08:05.991789 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls podName:d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d nodeName:}" failed. No retries permitted until 2026-07-13 21:08:06.991741682 +0000 UTC m=+862.720801560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls") pod "maas-api-6ff844954-j5st8" (UID: "d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d") : secret "maas-api-serving-cert" not found Jul 13 21:08:05.992464 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:05.992445 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:06.004329 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.004301 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvbb\" (UniqueName: \"kubernetes.io/projected/c7ba5f09-81dc-4ea0-b0e9-4439fc98594a-kube-api-access-jsvbb\") pod \"payload-processing-f9b8d9b9b-zjxzg\" (UID: \"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:06.093228 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.093138 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dc4ad8-de67-47d1-bd81-06a1998c80a6-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:06.093405 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.093297 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbvm\" (UniqueName: \"kubernetes.io/projected/85dc4ad8-de67-47d1-bd81-06a1998c80a6-kube-api-access-tnbvm\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:06.093967 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.093942 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dc4ad8-de67-47d1-bd81-06a1998c80a6-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:06.101782 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.101743 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbvm\" (UniqueName: \"kubernetes.io/projected/85dc4ad8-de67-47d1-bd81-06a1998c80a6-kube-api-access-tnbvm\") pod \"payload-pre-processing-74584bb9f8-pswft\" (UID: \"85dc4ad8-de67-47d1-bd81-06a1998c80a6\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:06.144904 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.144872 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:06.206372 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.206341 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:06.283207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.283179 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-zjxzg"] Jul 13 21:08:06.284904 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:08:06.284871 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ba5f09_81dc_4ea0_b0e9_4439fc98594a.slice/crio-43892d6792c0021cfd5fab7c2bb257c05b20cc3169e72165b663b78bed571448 WatchSource:0}: Error finding container 43892d6792c0021cfd5fab7c2bb257c05b20cc3169e72165b663b78bed571448: Status 404 returned error can't find the container with id 43892d6792c0021cfd5fab7c2bb257c05b20cc3169e72165b663b78bed571448 Jul 13 21:08:06.343642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.343599 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-pswft"] Jul 13 21:08:06.346318 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:08:06.346285 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dc4ad8_de67_47d1_bd81_06a1998c80a6.slice/crio-bab4c57386877022e5a4d291bdbb1ec495ef953b55220d7c3f128b8fde1552c7 WatchSource:0}: Error finding container bab4c57386877022e5a4d291bdbb1ec495ef953b55220d7c3f128b8fde1552c7: Status 404 returned error can't find the container with id bab4c57386877022e5a4d291bdbb1ec495ef953b55220d7c3f128b8fde1552c7 Jul 13 21:08:06.360046 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.360018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" event={"ID":"85dc4ad8-de67-47d1-bd81-06a1998c80a6","Type":"ContainerStarted","Data":"bab4c57386877022e5a4d291bdbb1ec495ef953b55220d7c3f128b8fde1552c7"} Jul 13 21:08:06.361167 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:06.361143 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" event={"ID":"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a","Type":"ContainerStarted","Data":"43892d6792c0021cfd5fab7c2bb257c05b20cc3169e72165b663b78bed571448"} Jul 13 21:08:07.008692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:07.008533 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:07.012826 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:07.012745 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d-maas-api-tls\") pod \"maas-api-6ff844954-j5st8\" (UID: \"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d\") " pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:07.128971 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:07.128914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:07.352323 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:07.352265 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ff844954-j5st8"] Jul 13 21:08:07.355001 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:08:07.354960 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71a8cd7_2a6d_4c3e_a761_55ac5ac76d0d.slice/crio-39b7482a90434797b733114f469eaaa893f5e6ab1437371e07a450cdafa242ee WatchSource:0}: Error finding container 39b7482a90434797b733114f469eaaa893f5e6ab1437371e07a450cdafa242ee: Status 404 returned error can't find the container with id 39b7482a90434797b733114f469eaaa893f5e6ab1437371e07a450cdafa242ee Jul 13 21:08:07.369654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:07.369616 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" event={"ID":"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d","Type":"ContainerStarted","Data":"39b7482a90434797b733114f469eaaa893f5e6ab1437371e07a450cdafa242ee"} Jul 13 21:08:08.377122 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.377081 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" event={"ID":"85dc4ad8-de67-47d1-bd81-06a1998c80a6","Type":"ContainerStarted","Data":"6cb810d8058f18d4d07edbc85dcd827c275fcb4086a83a28cc3f8cd95bd0ca83"} Jul 13 21:08:08.377604 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.377360 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:08.378910 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.378882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" event={"ID":"c7ba5f09-81dc-4ea0-b0e9-4439fc98594a","Type":"ContainerStarted","Data":"c55cd8c5f90197806e11b29082aa1111c661204bab446fb1077d1d67bfbae2cb"} Jul 13 21:08:08.379047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.379010 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:08.404787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.403689 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" podStartSLOduration=1.822799549 podStartE2EDuration="3.403638249s" totalImagesPullingTime="1.5808387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:08.401195673 +0000 UTC m=+864.130255574" watchObservedRunningTime="2026-07-13 21:08:08.403638249 +0000 UTC m=+864.132698130" Jul 13 21:08:08.418942 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:08.418886 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" podStartSLOduration=1.779686603 podStartE2EDuration="3.418864788s" totalImagesPullingTime="1.639178185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:08.418741139 +0000 UTC m=+864.147801040" watchObservedRunningTime="2026-07-13 21:08:08.418864788 +0000 UTC m=+864.147924688" Jul 13 21:08:10.409642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:10.409594 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" event={"ID":"d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d","Type":"ContainerStarted","Data":"ff279a182d8d86fc492fec5a5dffa4158eef5e97d159f947723fc1dc405dcf36"} Jul 13 21:08:10.410510 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:10.410485 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:10.426940 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:10.426702 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" podStartSLOduration=3.342614163 podStartE2EDuration="5.42668403s" totalImagesPullingTime="2.084069867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:10.425521143 +0000 UTC m=+866.154581043" watchObservedRunningTime="2026-07-13 21:08:10.42668403 +0000 UTC m=+866.155743932" Jul 13 21:08:14.434795 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.434745 2561 generic.go:356] "Generic (PLEG): container finished" podID="71ac7e7c-12a7-4050-84a0-023a08a6f1a2" containerID="e4c0d47a81ddf5ca3fd4efb5bce9bf6b070968aab3c9ad17464ecac521be1bb6" exitCode=0 Jul 13 21:08:14.435193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.434819 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" event={"ID":"71ac7e7c-12a7-4050-84a0-023a08a6f1a2","Type":"ContainerDied","Data":"e4c0d47a81ddf5ca3fd4efb5bce9bf6b070968aab3c9ad17464ecac521be1bb6"} Jul 13 21:08:14.435237 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.435219 2561 scope.go:117] "RemoveContainer" containerID="e4c0d47a81ddf5ca3fd4efb5bce9bf6b070968aab3c9ad17464ecac521be1bb6" Jul 13 21:08:14.436224 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.436196 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:08:14.436342 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.436300 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:08:14.436408 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.436350 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:08:14.929455 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:14.929425 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:08:15.441035 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:15.440998 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" event={"ID":"71ac7e7c-12a7-4050-84a0-023a08a6f1a2","Type":"ContainerStarted","Data":"8f717c0f61c440654aeec98e973f0f46d2434318f418b8d641f530ec004c0f66"} Jul 13 21:08:15.622018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:15.621981 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:08:15.623118 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:15.623093 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q8vsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 13 21:08:15.623256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:15.623138 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" podUID="71ac7e7c-12a7-4050-84a0-023a08a6f1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 13 21:08:16.621890 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:16.621854 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q8vsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 13 21:08:16.622283 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:16.621913 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" podUID="71ac7e7c-12a7-4050-84a0-023a08a6f1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 13 21:08:17.428594 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:17.428565 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6ff844954-j5st8" Jul 13 21:08:17.621513 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:17.621480 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q8vsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 13 21:08:17.621696 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:17.621532 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" podUID="71ac7e7c-12a7-4050-84a0-023a08a6f1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 13 21:08:18.621444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:18.621408 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-q8vsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 13 21:08:18.621875 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:18.621466 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" podUID="71ac7e7c-12a7-4050-84a0-023a08a6f1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 13 21:08:19.403204 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:19.403170 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-zjxzg" Jul 13 21:08:19.403391 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:19.403238 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-pswft" Jul 13 21:08:19.644510 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:19.644474 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:08:19.645006 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:19.644890 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:08:19.645808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:19.645785 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-q8vsw" Jul 13 21:08:39.051731 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.051649 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d"] Jul 13 21:08:39.057453 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.057432 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.059932 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.059914 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 21:08:39.060779 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.060744 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fb5tw\"" Jul 13 21:08:39.060779 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.060778 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 21:08:39.060972 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.060903 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 13 21:08:39.064982 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.064962 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d"] Jul 13 21:08:39.079952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.079921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.080145 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.079985 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.080145 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.080043 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xll\" (UniqueName: \"kubernetes.io/projected/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-kube-api-access-q2xll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.080490 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.080164 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.080490 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.080205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.080490 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.080228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181036 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181006 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181042 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xll\" (UniqueName: \"kubernetes.io/projected/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-kube-api-access-q2xll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181086 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181108 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181123 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181207 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181556 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181531 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181634 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.181634 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.181613 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.183559 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.183537 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.183724 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.183701 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.189979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.189960 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xll\" (UniqueName: \"kubernetes.io/projected/d62bab2a-cc2e-48fb-a46c-681f9cefa88f-kube-api-access-q2xll\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krl2d\" (UID: \"d62bab2a-cc2e-48fb-a46c-681f9cefa88f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.384341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.384302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:39.526027 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.526005 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d"] Jul 13 21:08:39.527955 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:08:39.527916 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62bab2a_cc2e_48fb_a46c_681f9cefa88f.slice/crio-e7db5fe62e997cefa4472d09cbb3aa4bcfb4bdf3ff2d4526dcd4571ec875beab WatchSource:0}: Error finding container e7db5fe62e997cefa4472d09cbb3aa4bcfb4bdf3ff2d4526dcd4571ec875beab: Status 404 returned error can't find the container with id e7db5fe62e997cefa4472d09cbb3aa4bcfb4bdf3ff2d4526dcd4571ec875beab Jul 13 21:08:39.568998 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:39.568966 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" event={"ID":"d62bab2a-cc2e-48fb-a46c-681f9cefa88f","Type":"ContainerStarted","Data":"e7db5fe62e997cefa4472d09cbb3aa4bcfb4bdf3ff2d4526dcd4571ec875beab"} Jul 13 21:08:41.582470 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:41.582420 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" event={"ID":"d62bab2a-cc2e-48fb-a46c-681f9cefa88f","Type":"ContainerStarted","Data":"4c14d204500e7a3e7f69dd820a3acbc46de650acae006f9752e91f28d3ff231d"} Jul 13 21:08:41.599988 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:41.599944 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" podStartSLOduration=1.409592492 podStartE2EDuration="2.599929389s" totalImagesPullingTime="1.190336897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:41.598919206 +0000 UTC m=+897.327979109" watchObservedRunningTime="2026-07-13 21:08:41.599929389 +0000 UTC m=+897.328989288" Jul 13 21:08:44.754632 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:44.754603 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:08:44.755189 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:44.754982 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:08:49.384597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:49.384554 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:49.385055 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:49.384629 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:49.397456 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:49.397431 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:49.635661 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:49.635577 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krl2d" Jul 13 21:08:58.252690 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.252652 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m"] Jul 13 21:08:58.257798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.257778 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.261365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.261346 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 13 21:08:58.272932 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.272910 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m"] Jul 13 21:08:58.352623 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352589 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.352814 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352628 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.352814 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2jq\" (UniqueName: \"kubernetes.io/projected/8546adf3-0b06-42c2-8375-c6eb6555bdc3-kube-api-access-kf2jq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.352814 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352717 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.352934 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.352934 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.352890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454104 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454104 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454109 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454326 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454229 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454326 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2jq\" (UniqueName: \"kubernetes.io/projected/8546adf3-0b06-42c2-8375-c6eb6555bdc3-kube-api-access-kf2jq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454326 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454477 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454530 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454494 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454584 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454552 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.454713 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.454687 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.456445 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.456425 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8546adf3-0b06-42c2-8375-c6eb6555bdc3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.456713 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.456697 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8546adf3-0b06-42c2-8375-c6eb6555bdc3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.462482 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.462459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2jq\" (UniqueName: \"kubernetes.io/projected/8546adf3-0b06-42c2-8375-c6eb6555bdc3-kube-api-access-kf2jq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m\" (UID: \"8546adf3-0b06-42c2-8375-c6eb6555bdc3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.568965 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.568879 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:08:58.723178 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.723144 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m"] Jul 13 21:08:58.730988 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:08:58.730958 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8546adf3_0b06_42c2_8375_c6eb6555bdc3.slice/crio-209ba503a594bb3ee079f09d17f34d0401156173f027248d84ade966dedf7c72 WatchSource:0}: Error finding container 209ba503a594bb3ee079f09d17f34d0401156173f027248d84ade966dedf7c72: Status 404 returned error can't find the container with id 209ba503a594bb3ee079f09d17f34d0401156173f027248d84ade966dedf7c72 Jul 13 21:08:58.733072 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:58.733047 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:08:59.676359 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:59.676320 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" event={"ID":"8546adf3-0b06-42c2-8375-c6eb6555bdc3","Type":"ContainerStarted","Data":"2843ad78ac9a8fa20c59a85c1eb66fbe693766ee463662d299cf1c5c9cb0bdf6"} Jul 13 21:08:59.676847 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:59.676377 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" event={"ID":"8546adf3-0b06-42c2-8375-c6eb6555bdc3","Type":"ContainerStarted","Data":"209ba503a594bb3ee079f09d17f34d0401156173f027248d84ade966dedf7c72"} Jul 13 21:08:59.694859 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:08:59.694805 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" podStartSLOduration=1.44278833 podStartE2EDuration="1.694790051s" totalImagesPullingTime="252.001721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:08:59.693043145 +0000 UTC m=+915.422103070" watchObservedRunningTime="2026-07-13 21:08:59.694790051 +0000 UTC m=+915.423849950" Jul 13 21:09:08.569081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:09:08.569041 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:09:08.569484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:09:08.569110 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:09:08.581522 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:09:08.581497 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:09:08.736378 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:09:08.736348 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m" Jul 13 21:10:10.013956 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.013910 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:10:10.035500 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.035456 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-775797f4d4-wcb72" podUID="cc627e29-9cbd-43b4-8736-89ae75456106" containerName="authorino" containerID="cri-o://0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516" gracePeriod=30 Jul 13 21:10:10.303747 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.303724 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:10:10.481554 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.481524 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert\") pod \"cc627e29-9cbd-43b4-8736-89ae75456106\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " Jul 13 21:10:10.481727 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.481565 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmvqw\" (UniqueName: \"kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw\") pod \"cc627e29-9cbd-43b4-8736-89ae75456106\" (UID: \"cc627e29-9cbd-43b4-8736-89ae75456106\") " Jul 13 21:10:10.483644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.483613 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw" (OuterVolumeSpecName: "kube-api-access-lmvqw") pod "cc627e29-9cbd-43b4-8736-89ae75456106" (UID: "cc627e29-9cbd-43b4-8736-89ae75456106"). InnerVolumeSpecName "kube-api-access-lmvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:10:10.494094 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.494063 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cc627e29-9cbd-43b4-8736-89ae75456106" (UID: "cc627e29-9cbd-43b4-8736-89ae75456106"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 21:10:10.583056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.582980 2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc627e29-9cbd-43b4-8736-89ae75456106-tls-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:10:10.583056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:10.583003 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmvqw\" (UniqueName: \"kubernetes.io/projected/cc627e29-9cbd-43b4-8736-89ae75456106-kube-api-access-lmvqw\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:10:11.041099 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.041061 2561 generic.go:356] "Generic (PLEG): container finished" podID="cc627e29-9cbd-43b4-8736-89ae75456106" containerID="0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516" exitCode=0 Jul 13 21:10:11.041492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.041115 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775797f4d4-wcb72" Jul 13 21:10:11.041492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.041108 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775797f4d4-wcb72" event={"ID":"cc627e29-9cbd-43b4-8736-89ae75456106","Type":"ContainerDied","Data":"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516"} Jul 13 21:10:11.041492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.041216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775797f4d4-wcb72" event={"ID":"cc627e29-9cbd-43b4-8736-89ae75456106","Type":"ContainerDied","Data":"e5a4cf89880a163e7f7946d6ce6c7c7762d92f4fb971987f25ae76c35a7bb24b"} Jul 13 21:10:11.041492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.041236 2561 scope.go:117] "RemoveContainer" containerID="0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516" Jul 13 21:10:11.050130 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.050113 2561 scope.go:117] "RemoveContainer" containerID="0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516" Jul 13 21:10:11.050403 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:10:11.050386 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516\": container with ID starting with 0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516 not found: ID does not exist" containerID="0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516" Jul 13 21:10:11.050461 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.050410 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516"} err="failed to get container status \"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516\": rpc error: code = NotFound desc = could not find container \"0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516\": container with ID starting with 0e4138d37765618112954fcdbcc4f7308f8f73e7464536b55d4262809f24b516 not found: ID does not exist" Jul 13 21:10:11.058397 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.058377 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:10:11.061269 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:11.061243 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-775797f4d4-wcb72"] Jul 13 21:10:12.834630 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:10:12.834596 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc627e29-9cbd-43b4-8736-89ae75456106" path="/var/lib/kubelet/pods/cc627e29-9cbd-43b4-8736-89ae75456106/volumes" Jul 13 21:11:23.481616 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.481571 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f6dc4cf54-x9twb"] Jul 13 21:11:23.482134 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.482030 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc627e29-9cbd-43b4-8736-89ae75456106" containerName="authorino" Jul 13 21:11:23.482134 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.482042 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc627e29-9cbd-43b4-8736-89ae75456106" containerName="authorino" Jul 13 21:11:23.482134 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.482106 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc627e29-9cbd-43b4-8736-89ae75456106" containerName="authorino" Jul 13 21:11:23.485503 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.485482 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.488708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.488685 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 21:11:23.488708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.488700 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 21:11:23.488884 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.488861 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k4bbc\"" Jul 13 21:11:23.493734 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.493704 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6dc4cf54-x9twb"] Jul 13 21:11:23.530455 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.530425 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmdl\" (UniqueName: \"kubernetes.io/projected/edf8fe23-9275-461c-b729-16bff5ef3714-kube-api-access-czmdl\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.530587 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.530459 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf8fe23-9275-461c-b729-16bff5ef3714-webhook-cert\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.631511 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.631473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmdl\" (UniqueName: \"kubernetes.io/projected/edf8fe23-9275-461c-b729-16bff5ef3714-kube-api-access-czmdl\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.631694 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.631517 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf8fe23-9275-461c-b729-16bff5ef3714-webhook-cert\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.634006 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.633973 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edf8fe23-9275-461c-b729-16bff5ef3714-webhook-cert\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.641035 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.641011 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmdl\" (UniqueName: \"kubernetes.io/projected/edf8fe23-9275-461c-b729-16bff5ef3714-kube-api-access-czmdl\") pod \"maas-controller-6f6dc4cf54-x9twb\" (UID: \"edf8fe23-9275-461c-b729-16bff5ef3714\") " pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:23.797309 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:23.797220 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:24.137891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:24.137862 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f6dc4cf54-x9twb"] Jul 13 21:11:24.139804 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:11:24.139772 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf8fe23_9275_461c_b729_16bff5ef3714.slice/crio-b04df35ceaac681f430ede06fd6282de02db6c61e6716395aafa6ca5071616aa WatchSource:0}: Error finding container b04df35ceaac681f430ede06fd6282de02db6c61e6716395aafa6ca5071616aa: Status 404 returned error can't find the container with id b04df35ceaac681f430ede06fd6282de02db6c61e6716395aafa6ca5071616aa Jul 13 21:11:24.411904 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:24.411822 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" event={"ID":"edf8fe23-9275-461c-b729-16bff5ef3714","Type":"ContainerStarted","Data":"b04df35ceaac681f430ede06fd6282de02db6c61e6716395aafa6ca5071616aa"} Jul 13 21:11:25.418809 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:25.418772 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" event={"ID":"edf8fe23-9275-461c-b729-16bff5ef3714","Type":"ContainerStarted","Data":"d09936e42c3ab3122ed607147858ded4e08c85e246d49072d18ced2cf6661b6e"} Jul 13 21:11:25.419197 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:25.418871 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:11:25.440530 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:25.440477 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" podStartSLOduration=1.950436431 podStartE2EDuration="2.44046272s" totalImagesPullingTime="490.026289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:11:25.438636145 +0000 UTC m=+1061.167696044" watchObservedRunningTime="2026-07-13 21:11:25.44046272 +0000 UTC m=+1061.169522619" Jul 13 21:11:36.426538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:11:36.426503 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f6dc4cf54-x9twb" Jul 13 21:13:44.785392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:13:44.785360 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:13:44.788013 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:13:44.786109 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:15:00.142564 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.142484 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n"] Jul 13 21:15:00.146136 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.146117 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" Jul 13 21:15:00.148851 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.148827 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:15:00.155735 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.155718 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" Jul 13 21:15:00.164277 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.164257 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n"] Jul 13 21:15:00.287678 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.287654 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n"] Jul 13 21:15:00.290173 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:15:00.290127 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655ab8b2_e8be_4618_93fb_0cecd692913b.slice/crio-ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311 WatchSource:0}: Error finding container ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311: Status 404 returned error can't find the container with id ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311 Jul 13 21:15:00.292130 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.292112 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:15:00.506217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:00.506123 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" event={"ID":"655ab8b2-e8be-4618-93fb-0cecd692913b","Type":"ContainerStarted","Data":"ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311"} Jul 13 21:15:05.533925 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:05.533883 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" event={"ID":"655ab8b2-e8be-4618-93fb-0cecd692913b","Type":"ContainerStarted","Data":"41f3b02c928eeb01fed77e1345d038f95bf67c32a49e99270658c1865be2eef5"} Jul 13 21:15:05.554114 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:05.554061 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" podStartSLOduration=0.992609117 podStartE2EDuration="5.554046099s" totalImagesPullingTime="4.561436982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:15:05.552228166 +0000 UTC m=+1281.281288067" watchObservedRunningTime="2026-07-13 21:15:05.554046099 +0000 UTC m=+1281.283105999" Jul 13 21:15:06.539865 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:06.539829 2561 generic.go:356] "Generic (PLEG): container finished" podID="655ab8b2-e8be-4618-93fb-0cecd692913b" containerID="41f3b02c928eeb01fed77e1345d038f95bf67c32a49e99270658c1865be2eef5" exitCode=0 Jul 13 21:15:06.540256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:06.539916 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" event={"ID":"655ab8b2-e8be-4618-93fb-0cecd692913b","Type":"ContainerDied","Data":"41f3b02c928eeb01fed77e1345d038f95bf67c32a49e99270658c1865be2eef5"} Jul 13 21:15:07.680142 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:07.680117 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" Jul 13 21:15:08.550257 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:08.550216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" event={"ID":"655ab8b2-e8be-4618-93fb-0cecd692913b","Type":"ContainerDied","Data":"ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311"} Jul 13 21:15:08.550438 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:08.550275 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea5f6a7f78b703b1ba83f995fdd830e4df9fcb92999dafa4fa4ec7b551a38311" Jul 13 21:15:08.550438 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:15:08.550229 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732955-9hq6n" Jul 13 21:16:13.806189 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:13.806153 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:16:13.806779 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:13.806355 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" podUID="415778b1-ffdd-4815-bb3f-6690d89ab180" containerName="manager" containerID="cri-o://453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3" gracePeriod=10 Jul 13 21:16:16.865151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.865125 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:16:16.894034 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.894002 2561 generic.go:356] "Generic (PLEG): container finished" podID="415778b1-ffdd-4815-bb3f-6690d89ab180" containerID="453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3" exitCode=0 Jul 13 21:16:16.894213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.894046 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" Jul 13 21:16:16.894213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.894068 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" event={"ID":"415778b1-ffdd-4815-bb3f-6690d89ab180","Type":"ContainerDied","Data":"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3"} Jul 13 21:16:16.894213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.894111 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml" event={"ID":"415778b1-ffdd-4815-bb3f-6690d89ab180","Type":"ContainerDied","Data":"75693a763ad476ab4feb2d47804d94dac95404b23f4fea0936506815ff6bca0b"} Jul 13 21:16:16.894213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.894131 2561 scope.go:117] "RemoveContainer" containerID="453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3" Jul 13 21:16:16.909070 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.909045 2561 scope.go:117] "RemoveContainer" containerID="453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3" Jul 13 21:16:16.909665 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:16:16.909634 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3\": container with ID starting with 453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3 not found: ID does not exist" containerID="453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3" Jul 13 21:16:16.909799 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.909676 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3"} err="failed to get container status \"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3\": rpc error: code = NotFound desc = could not find container \"453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3\": container with ID starting with 453e5831d1c19298cf2bf887ed4892eed6f35350384dbb885f4f616f415253a3 not found: ID does not exist" Jul 13 21:16:16.978665 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.978588 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume\") pod \"415778b1-ffdd-4815-bb3f-6690d89ab180\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " Jul 13 21:16:16.978665 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.978649 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b\") pod \"415778b1-ffdd-4815-bb3f-6690d89ab180\" (UID: \"415778b1-ffdd-4815-bb3f-6690d89ab180\") " Jul 13 21:16:16.979059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.979037 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "415778b1-ffdd-4815-bb3f-6690d89ab180" (UID: "415778b1-ffdd-4815-bb3f-6690d89ab180"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:16:16.980699 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:16.980673 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b" (OuterVolumeSpecName: "kube-api-access-wfs9b") pod "415778b1-ffdd-4815-bb3f-6690d89ab180" (UID: "415778b1-ffdd-4815-bb3f-6690d89ab180"). InnerVolumeSpecName "kube-api-access-wfs9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:16:17.079313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:17.079281 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/415778b1-ffdd-4815-bb3f-6690d89ab180-extensions-socket-volume\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:16:17.079313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:17.079310 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfs9b\" (UniqueName: \"kubernetes.io/projected/415778b1-ffdd-4815-bb3f-6690d89ab180-kube-api-access-wfs9b\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:16:17.217179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:17.217151 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:16:17.224903 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:17.224877 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fskml"] Jul 13 21:16:18.836854 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:16:18.836813 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="415778b1-ffdd-4815-bb3f-6690d89ab180" path="/var/lib/kubelet/pods/415778b1-ffdd-4815-bb3f-6690d89ab180/volumes" Jul 13 21:17:14.941179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941138 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46"] Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941527 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655ab8b2-e8be-4618-93fb-0cecd692913b" containerName="cleanup" Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941539 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="655ab8b2-e8be-4618-93fb-0cecd692913b" containerName="cleanup" Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941552 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="415778b1-ffdd-4815-bb3f-6690d89ab180" containerName="manager" Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941558 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="415778b1-ffdd-4815-bb3f-6690d89ab180" containerName="manager" Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941628 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="415778b1-ffdd-4815-bb3f-6690d89ab180" containerName="manager" Jul 13 21:17:14.941692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.941650 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="655ab8b2-e8be-4618-93fb-0cecd692913b" containerName="cleanup" Jul 13 21:17:14.944481 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.944463 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:14.947008 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.946987 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nk4b5\"" Jul 13 21:17:14.963828 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.963798 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46"] Jul 13 21:17:14.989385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.989361 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a268a4a4-a21a-4abf-9ab5-f18760002309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:14.989516 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:14.989424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92cn4\" (UniqueName: \"kubernetes.io/projected/a268a4a4-a21a-4abf-9ab5-f18760002309-kube-api-access-92cn4\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.090299 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.090263 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a268a4a4-a21a-4abf-9ab5-f18760002309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.090463 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.090383 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92cn4\" (UniqueName: \"kubernetes.io/projected/a268a4a4-a21a-4abf-9ab5-f18760002309-kube-api-access-92cn4\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.090639 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.090609 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a268a4a4-a21a-4abf-9ab5-f18760002309-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.098381 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.098358 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92cn4\" (UniqueName: \"kubernetes.io/projected/a268a4a4-a21a-4abf-9ab5-f18760002309-kube-api-access-92cn4\") pod \"kuadrant-operator-controller-manager-86878c769f-2gw46\" (UID: \"a268a4a4-a21a-4abf-9ab5-f18760002309\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.254974 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.254873 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:15.392905 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:15.392874 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46"] Jul 13 21:17:15.396173 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:17:15.396135 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda268a4a4_a21a_4abf_9ab5_f18760002309.slice/crio-337eedbd9fff4f4b8d11ebc214dba687004ea83bbfc4a77bbd26068ab0ffcfbe WatchSource:0}: Error finding container 337eedbd9fff4f4b8d11ebc214dba687004ea83bbfc4a77bbd26068ab0ffcfbe: Status 404 returned error can't find the container with id 337eedbd9fff4f4b8d11ebc214dba687004ea83bbfc4a77bbd26068ab0ffcfbe Jul 13 21:17:16.212636 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:16.212602 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" event={"ID":"a268a4a4-a21a-4abf-9ab5-f18760002309","Type":"ContainerStarted","Data":"38c5c6bda5ad039edc09bb8b30fd85896bf7ce41295fa0b6cf0577d2723cc9cd"} Jul 13 21:17:16.213152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:16.212667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" event={"ID":"a268a4a4-a21a-4abf-9ab5-f18760002309","Type":"ContainerStarted","Data":"337eedbd9fff4f4b8d11ebc214dba687004ea83bbfc4a77bbd26068ab0ffcfbe"} Jul 13 21:17:16.213152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:16.212742 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:17:16.236741 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:16.236695 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" podStartSLOduration=2.236680822 podStartE2EDuration="2.236680822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:17:16.236128958 +0000 UTC m=+1411.965188897" watchObservedRunningTime="2026-07-13 21:17:16.236680822 +0000 UTC m=+1411.965740722" Jul 13 21:17:27.220634 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:17:27.220603 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2gw46" Jul 13 21:18:44.816450 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:18:44.816344 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:18:44.819685 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:18:44.818534 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:23:36.336324 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.336289 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:23:36.340024 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.340002 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.342634 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.342608 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-19927e2d-openshift-default-dockercfg-rdn7k\"" Jul 13 21:23:36.365167 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.365145 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:23:36.518820 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.518783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.518993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.518837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.518993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.518875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.518993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.518900 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.518993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.518916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.519149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.519038 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq4n\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.519149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.519068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.519149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.519090 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.519149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.519112 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619621 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619589 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619800 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619624 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619800 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq4n\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619800 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619932 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619871 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.619932 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620016 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619951 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620016 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.619985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620049 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620096 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620288 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620194 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620410 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620391 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.620675 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.620654 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.622093 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.622075 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.622440 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.622418 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.629577 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.629553 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq4n\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.629908 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.629892 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token\") pod \"e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.651660 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.651642 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:36.794781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.794721 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:23:36.796790 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:23:36.796744 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d650c0_d180_410f_bcd7_58c64e15c648.slice/crio-8c25abfcb32c95abcf5388a31e4dc75e4cda518c8294fb8e5d87f8acc359a867 WatchSource:0}: Error finding container 8c25abfcb32c95abcf5388a31e4dc75e4cda518c8294fb8e5d87f8acc359a867: Status 404 returned error can't find the container with id 8c25abfcb32c95abcf5388a31e4dc75e4cda518c8294fb8e5d87f8acc359a867 Jul 13 21:23:36.798991 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.798950 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:36.799076 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.799059 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:36.799116 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:36.799105 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:37.161096 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:37.161059 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerStarted","Data":"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8"} Jul 13 21:23:37.161247 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:37.161124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerStarted","Data":"8c25abfcb32c95abcf5388a31e4dc75e4cda518c8294fb8e5d87f8acc359a867"} Jul 13 21:23:37.188357 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:37.188306 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podStartSLOduration=1.188288194 podStartE2EDuration="1.188288194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:23:37.185601891 +0000 UTC m=+1792.914661792" watchObservedRunningTime="2026-07-13 21:23:37.188288194 +0000 UTC m=+1792.917348096" Jul 13 21:23:37.652376 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:37.652346 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:37.657290 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:37.657266 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:38.167048 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:38.167014 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:38.167980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:38.167962 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:42.570559 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.570522 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:23:42.575877 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.575855 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:23:42.578940 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.578915 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:23:42.588877 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.588843 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:23:42.672688 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.672650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd\" (UID: \"60972231-4cc8-4107-863a-a05b6191f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:23:42.773550 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.773510 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd\" (UID: \"60972231-4cc8-4107-863a-a05b6191f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:23:42.774265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.774247 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd\" (UID: \"60972231-4cc8-4107-863a-a05b6191f88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:23:42.891217 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:42.891180 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:23:43.069380 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:43.069304 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:23:43.073482 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:23:43.073449 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60972231_4cc8_4107_863a_a05b6191f88a.slice/crio-bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601 WatchSource:0}: Error finding container bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601: Status 404 returned error can't find the container with id bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601 Jul 13 21:23:43.211805 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:43.204249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerStarted","Data":"bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601"} Jul 13 21:23:44.214253 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:44.214215 2561 generic.go:356] "Generic (PLEG): container finished" podID="60972231-4cc8-4107-863a-a05b6191f88a" containerID="89a542ef8b03e760330717fc7f4c75889d5a4ede7b9f20259cd3df64dc475a2b" exitCode=7 Jul 13 21:23:44.214673 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:44.214296 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerDied","Data":"89a542ef8b03e760330717fc7f4c75889d5a4ede7b9f20259cd3df64dc475a2b"} Jul 13 21:23:44.214673 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:44.214466 2561 scope.go:117] "RemoveContainer" containerID="89a542ef8b03e760330717fc7f4c75889d5a4ede7b9f20259cd3df64dc475a2b" Jul 13 21:23:44.849518 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:44.849497 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:23:44.859886 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:44.851370 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:23:45.221885 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:45.221855 2561 generic.go:356] "Generic (PLEG): container finished" podID="60972231-4cc8-4107-863a-a05b6191f88a" containerID="39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47" exitCode=7 Jul 13 21:23:45.222344 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:45.221944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerDied","Data":"39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47"} Jul 13 21:23:45.222344 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:45.222007 2561 scope.go:117] "RemoveContainer" containerID="89a542ef8b03e760330717fc7f4c75889d5a4ede7b9f20259cd3df64dc475a2b" Jul 13 21:23:45.222344 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:45.222231 2561 scope.go:117] "RemoveContainer" containerID="39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47" Jul 13 21:23:45.222513 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:23:45.222488 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd_odh-ai-gateway-infra(60972231-4cc8-4107-863a-a05b6191f88a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" podUID="60972231-4cc8-4107-863a-a05b6191f88a" Jul 13 21:23:46.230375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:46.230344 2561 scope.go:117] "RemoveContainer" containerID="39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47" Jul 13 21:23:46.230794 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:23:46.230609 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd_odh-ai-gateway-infra(60972231-4cc8-4107-863a-a05b6191f88a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" podUID="60972231-4cc8-4107-863a-a05b6191f88a" Jul 13 21:23:48.245241 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.245212 2561 generic.go:356] "Generic (PLEG): container finished" podID="31d650c0-d180-410f-bcd7-58c64e15c648" containerID="0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8" exitCode=0 Jul 13 21:23:48.245586 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.245251 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerDied","Data":"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8"} Jul 13 21:23:48.245711 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.245692 2561 scope.go:117] "RemoveContainer" containerID="0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8" Jul 13 21:23:48.246682 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.246656 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:48.246729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.246721 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:48.246789 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:48.246774 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:23:49.252611 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:49.252572 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerStarted","Data":"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b"} Jul 13 21:23:49.651873 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:49.651834 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:49.652653 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:49.652628 2561 patch_prober.go:28] interesting pod/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 13 21:23:49.652798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:49.652678 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 13 21:23:50.652427 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:50.652395 2561 patch_prober.go:28] interesting pod/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 13 21:23:50.652844 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:50.652448 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 13 21:23:51.652539 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:51.652497 2561 patch_prober.go:28] interesting pod/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 13 21:23:51.653032 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:51.652545 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 13 21:23:52.652565 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:52.652526 2561 patch_prober.go:28] interesting pod/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 13 21:23:52.653011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:52.652582 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 13 21:23:53.167174 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:53.167128 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:53.656931 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:53.656902 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:23:53.657741 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:23:53.657723 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:24:00.829190 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:00.829163 2561 scope.go:117] "RemoveContainer" containerID="39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47" Jul 13 21:24:01.313721 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:01.313685 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerStarted","Data":"3f640f0326df4adc65c0b85a71f814f4278ceb77a9a38ae51f46bb58ee07e4e7"} Jul 13 21:24:01.330492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:01.330440 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" podStartSLOduration=19.330425112 podStartE2EDuration="19.330425112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:24:01.329127139 +0000 UTC m=+1817.058187070" watchObservedRunningTime="2026-07-13 21:24:01.330425112 +0000 UTC m=+1817.059485012" Jul 13 21:24:02.321183 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:02.321148 2561 generic.go:356] "Generic (PLEG): container finished" podID="60972231-4cc8-4107-863a-a05b6191f88a" containerID="3f640f0326df4adc65c0b85a71f814f4278ceb77a9a38ae51f46bb58ee07e4e7" exitCode=0 Jul 13 21:24:02.321183 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:02.321186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerDied","Data":"3f640f0326df4adc65c0b85a71f814f4278ceb77a9a38ae51f46bb58ee07e4e7"} Jul 13 21:24:02.321666 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:02.321234 2561 scope.go:117] "RemoveContainer" containerID="39e9d5581064058c8961b69af5ba4f4a91799c1d235ab1e948f68584fb899d47" Jul 13 21:24:02.346088 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:02.346060 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:24:03.467568 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:03.467545 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:24:03.562647 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:03.562617 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca\") pod \"60972231-4cc8-4107-863a-a05b6191f88a\" (UID: \"60972231-4cc8-4107-863a-a05b6191f88a\") " Jul 13 21:24:03.562979 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:03.562957 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60972231-4cc8-4107-863a-a05b6191f88a" (UID: "60972231-4cc8-4107-863a-a05b6191f88a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:24:03.663372 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:03.663347 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60972231-4cc8-4107-863a-a05b6191f88a-maas-api-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:24:04.333916 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.333890 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" Jul 13 21:24:04.334089 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.333935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd" event={"ID":"60972231-4cc8-4107-863a-a05b6191f88a","Type":"ContainerDied","Data":"bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601"} Jul 13 21:24:04.334089 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.333988 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc26366380e9a333a2e425a98904a529f5a8a655b03bb67b376d668616d9b601" Jul 13 21:24:04.366247 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.366224 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:24:04.369775 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.369737 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19927e2d-j6tdd"] Jul 13 21:24:04.834717 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:24:04.834678 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60972231-4cc8-4107-863a-a05b6191f88a" path="/var/lib/kubelet/pods/60972231-4cc8-4107-863a-a05b6191f88a/volumes" Jul 13 21:25:07.860861 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:07.860826 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:25:08.418628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.418596 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:08.419262 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419245 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419318 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419269 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419318 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419315 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419324 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419422 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419413 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419455 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419427 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.419455 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.419436 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:08.422932 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.422912 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.426078 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.426051 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a6867856-gw-openshift-default-dockercfg-tfhr8\"" Jul 13 21:25:08.442232 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.442202 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:08.553548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553513 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.553786 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.553786 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.553786 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553722 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.553786 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.554037 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.554037 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.554037 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553941 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.554037 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.553983 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbngk\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.654733 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.654694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.654945 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.654786 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.654945 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.654829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbngk\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655022 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.654968 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655022 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.654998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655132 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655027 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655132 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655080 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655132 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655278 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655163 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655278 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655215 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655471 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655444 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655471 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655467 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655630 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.655866 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.655849 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.657330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.657307 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.657470 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.657450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.664917 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.664879 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" containerID="cri-o://41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b" gracePeriod=30 Jul 13 21:25:08.665893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.665871 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.666517 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.666492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbngk\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk\") pod \"e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.734399 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.734302 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:08.882406 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.882372 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:08.884603 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:25:08.884575 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2754e399_c028_40d9_97b5_2c7b3d095a34.slice/crio-6df4475e2177b3ccb1df003b59845624152a966b9d83df518d7afa9315c2c2de WatchSource:0}: Error finding container 6df4475e2177b3ccb1df003b59845624152a966b9d83df518d7afa9315c2c2de: Status 404 returned error can't find the container with id 6df4475e2177b3ccb1df003b59845624152a966b9d83df518d7afa9315c2c2de Jul 13 21:25:08.887416 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.887375 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:08.887508 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.887466 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:08.887546 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:08.887514 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:09.671287 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:09.671246 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerStarted","Data":"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6"} Jul 13 21:25:09.671447 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:09.671307 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerStarted","Data":"6df4475e2177b3ccb1df003b59845624152a966b9d83df518d7afa9315c2c2de"} Jul 13 21:25:09.696364 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:09.696299 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podStartSLOduration=1.696274252 podStartE2EDuration="1.696274252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:25:09.693468328 +0000 UTC m=+1885.422528229" watchObservedRunningTime="2026-07-13 21:25:09.696274252 +0000 UTC m=+1885.425334152" Jul 13 21:25:09.735164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:09.735131 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:09.740184 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:09.740160 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:10.676181 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:10.676144 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:10.676990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:10.676970 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:13.930744 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:13.930683 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:25:14.105302 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105252 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rq4n\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105475 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105322 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105475 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105425 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105481 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105522 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105543 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105565 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105644 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105613 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105669 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy\") pod \"31d650c0-d180-410f-bcd7-58c64e15c648\" (UID: \"31d650c0-d180-410f-bcd7-58c64e15c648\") " Jul 13 21:25:14.105990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105699 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:14.105990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105823 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:14.105990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105940 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:14.105990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.105972 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:25:14.106308 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.106060 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.106308 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.106078 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d650c0-d180-410f-bcd7-58c64e15c648-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.106308 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.106095 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.106308 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.106115 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.106507 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.106421 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data" (OuterVolumeSpecName: "istio-data") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:14.107880 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.107829 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:25:14.108007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.107938 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token" (OuterVolumeSpecName: "istio-token") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:25:14.108197 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.108181 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:14.108308 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.108286 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n" (OuterVolumeSpecName: "kube-api-access-9rq4n") pod "31d650c0-d180-410f-bcd7-58c64e15c648" (UID: "31d650c0-d180-410f-bcd7-58c64e15c648"). InnerVolumeSpecName "kube-api-access-9rq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:25:14.207275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.207198 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.207275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.207227 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rq4n\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-kube-api-access-9rq4n\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.207275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.207239 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d650c0-d180-410f-bcd7-58c64e15c648-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.207275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.207249 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d650c0-d180-410f-bcd7-58c64e15c648-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.207275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.207257 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d650c0-d180-410f-bcd7-58c64e15c648-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:14.701586 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.701554 2561 generic.go:356] "Generic (PLEG): container finished" podID="31d650c0-d180-410f-bcd7-58c64e15c648" containerID="41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b" exitCode=0 Jul 13 21:25:14.701829 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.701596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerDied","Data":"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b"} Jul 13 21:25:14.701829 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.701606 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" Jul 13 21:25:14.701829 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.701648 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj" event={"ID":"31d650c0-d180-410f-bcd7-58c64e15c648","Type":"ContainerDied","Data":"8c25abfcb32c95abcf5388a31e4dc75e4cda518c8294fb8e5d87f8acc359a867"} Jul 13 21:25:14.701829 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.701675 2561 scope.go:117] "RemoveContainer" containerID="41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b" Jul 13 21:25:14.711896 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.711879 2561 scope.go:117] "RemoveContainer" containerID="0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8" Jul 13 21:25:14.720233 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.720215 2561 scope.go:117] "RemoveContainer" containerID="41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b" Jul 13 21:25:14.720465 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:14.720445 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b\": container with ID starting with 41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b not found: ID does not exist" containerID="41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b" Jul 13 21:25:14.720516 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.720474 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b"} err="failed to get container status \"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b\": rpc error: code = NotFound desc = could not find container \"41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b\": container with ID starting with 41b8224985a861430e1df779a1b34e18a01961d4f30d427612b34a2fc04b276b not found: ID does not exist" Jul 13 21:25:14.720516 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.720492 2561 scope.go:117] "RemoveContainer" containerID="0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8" Jul 13 21:25:14.720699 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:14.720685 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8\": container with ID starting with 0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8 not found: ID does not exist" containerID="0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8" Jul 13 21:25:14.720746 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.720702 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8"} err="failed to get container status \"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8\": rpc error: code = NotFound desc = could not find container \"0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8\": container with ID starting with 0fd888cb9d59b27c02db2dcf9244b5855fdf72ed621d92851aaed18a749496f8 not found: ID does not exist" Jul 13 21:25:14.724289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.724268 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:25:14.728993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.728970 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-19927e2d-openshift-default-7f945d64bf-rx9sj"] Jul 13 21:25:14.845625 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:14.845586 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" path="/var/lib/kubelet/pods/31d650c0-d180-410f-bcd7-58c64e15c648/volumes" Jul 13 21:25:25.678047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.678012 2561 patch_prober.go:28] interesting pod/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 13 21:25:25.678460 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.678065 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 13 21:25:25.762209 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.762172 2561 generic.go:356] "Generic (PLEG): container finished" podID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerID="c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6" exitCode=0 Jul 13 21:25:25.762373 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.762249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerDied","Data":"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6"} Jul 13 21:25:25.762659 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.762645 2561 scope.go:117] "RemoveContainer" containerID="c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6" Jul 13 21:25:25.763701 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.763676 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:25.763798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.763781 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:25.763845 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:25.763823 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:25:26.768787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:26.768732 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerStarted","Data":"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc"} Jul 13 21:25:27.735348 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:27.735312 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:27.735607 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:27.735588 2561 patch_prober.go:28] interesting pod/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 13 21:25:27.735659 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:27.735643 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 13 21:25:28.734697 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:28.734664 2561 patch_prober.go:28] interesting pod/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 13 21:25:28.735097 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:28.734727 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 13 21:25:29.734808 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:29.734742 2561 patch_prober.go:28] interesting pod/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 13 21:25:29.735218 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:29.734833 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 13 21:25:30.738820 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:30.738790 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:30.739231 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:30.739032 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:30.739813 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:30.739796 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:39.453322 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:39.453275 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:39.453860 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:39.453550 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" containerID="cri-o://40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc" gracePeriod=30 Jul 13 21:25:44.717175 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.717117 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:44.779794 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779741 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779839 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbngk\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779865 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779903 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779927 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779949 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.779984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.779978 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.780313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780016 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.780313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780100 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo\") pod \"2754e399-c028-40d9-97b5-2c7b3d095a34\" (UID: \"2754e399-c028-40d9-97b5-2c7b3d095a34\") " Jul 13 21:25:44.780313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780165 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:44.780462 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780323 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:44.780520 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780473 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.780520 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780492 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.780520 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780496 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:25:44.780684 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780661 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:44.780743 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.780687 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data" (OuterVolumeSpecName: "istio-data") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:44.782996 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.782970 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:25:44.783159 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.783037 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:25:44.783507 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.783478 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token" (OuterVolumeSpecName: "istio-token") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:25:44.785518 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.785488 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk" (OuterVolumeSpecName: "kube-api-access-vbngk") pod "2754e399-c028-40d9-97b5-2c7b3d095a34" (UID: "2754e399-c028-40d9-97b5-2c7b3d095a34"). InnerVolumeSpecName "kube-api-access-vbngk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:25:44.872669 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.872635 2561 generic.go:356] "Generic (PLEG): container finished" podID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerID="40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc" exitCode=0 Jul 13 21:25:44.872831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.872682 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" Jul 13 21:25:44.872831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.872713 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerDied","Data":"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc"} Jul 13 21:25:44.872831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.872788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj" event={"ID":"2754e399-c028-40d9-97b5-2c7b3d095a34","Type":"ContainerDied","Data":"6df4475e2177b3ccb1df003b59845624152a966b9d83df518d7afa9315c2c2de"} Jul 13 21:25:44.872831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.872812 2561 scope.go:117] "RemoveContainer" containerID="40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc" Jul 13 21:25:44.881477 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.881459 2561 scope.go:117] "RemoveContainer" containerID="c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6" Jul 13 21:25:44.881965 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.881943 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbngk\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-kube-api-access-vbngk\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.881970 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.881985 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.881998 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2754e399-c028-40d9-97b5-2c7b3d095a34-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.882011 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.882023 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.882073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.882036 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2754e399-c028-40d9-97b5-2c7b3d095a34-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:25:44.889780 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.889764 2561 scope.go:117] "RemoveContainer" containerID="40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc" Jul 13 21:25:44.890039 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:44.890022 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc\": container with ID starting with 40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc not found: ID does not exist" containerID="40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc" Jul 13 21:25:44.890110 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.890046 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc"} err="failed to get container status \"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc\": rpc error: code = NotFound desc = could not find container \"40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc\": container with ID starting with 40896c18ff99160a592508ae5c851ee57f01447192cfc3044def97e5d1471ecc not found: ID does not exist" Jul 13 21:25:44.890110 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.890062 2561 scope.go:117] "RemoveContainer" containerID="c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6" Jul 13 21:25:44.890313 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:44.890297 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6\": container with ID starting with c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6 not found: ID does not exist" containerID="c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6" Jul 13 21:25:44.890352 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.890318 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6"} err="failed to get container status \"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6\": rpc error: code = NotFound desc = could not find container \"c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6\": container with ID starting with c6c6ab151e39bc856d03f92cd3cfdb1edccd30d74c83332241f695eaa2f36cb6 not found: ID does not exist" Jul 13 21:25:44.894396 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.894374 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:44.906643 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:44.906622 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a6867856-gw-openshift-default-75d697944c-ndmqj"] Jul 13 21:25:45.939642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.939605 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:25:45.940051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940037 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:45.940051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940052 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="60972231-4cc8-4107-863a-a05b6191f88a" containerName="revoke-keys" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940061 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940067 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940075 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940080 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940091 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:25:45.940128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940096 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:25:45.940300 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940170 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:25:45.940300 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940178 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:25:45.940300 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.940185 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:25:45.943224 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.943205 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:25:45.945718 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.945697 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:25:45.952253 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:45.952230 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:25:46.095188 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.095086 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9\" (UID: \"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:25:46.195954 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.195911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9\" (UID: \"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:25:46.196531 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.196508 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9\" (UID: \"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:25:46.256419 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.256393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:25:46.388903 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.388874 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:25:46.389922 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:25:46.389898 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffeb5b4_0aed_476d_b569_5c48cb22a0f4.slice/crio-bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c WatchSource:0}: Error finding container bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c: Status 404 returned error can't find the container with id bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c Jul 13 21:25:46.836867 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.836834 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" path="/var/lib/kubelet/pods/2754e399-c028-40d9-97b5-2c7b3d095a34/volumes" Jul 13 21:25:46.884831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.884793 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerStarted","Data":"5e6e5d62d6652e736eaa4845f9cfd9289d74e17fa8c30906a385c6197f322dfb"} Jul 13 21:25:46.885030 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.884855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerStarted","Data":"bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c"} Jul 13 21:25:46.908011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:46.907962 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" podStartSLOduration=1.907944794 podStartE2EDuration="1.907944794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:25:46.905503231 +0000 UTC m=+1922.634563131" watchObservedRunningTime="2026-07-13 21:25:46.907944794 +0000 UTC m=+1922.637004694" Jul 13 21:25:47.892210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:47.892169 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerID="5e6e5d62d6652e736eaa4845f9cfd9289d74e17fa8c30906a385c6197f322dfb" exitCode=7 Jul 13 21:25:47.892725 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:47.892256 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerDied","Data":"5e6e5d62d6652e736eaa4845f9cfd9289d74e17fa8c30906a385c6197f322dfb"} Jul 13 21:25:47.892725 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:47.892433 2561 scope.go:117] "RemoveContainer" containerID="5e6e5d62d6652e736eaa4845f9cfd9289d74e17fa8c30906a385c6197f322dfb" Jul 13 21:25:48.898664 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:48.898632 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerID="ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1" exitCode=7 Jul 13 21:25:48.899221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:48.898702 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerDied","Data":"ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1"} Jul 13 21:25:48.899221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:48.898742 2561 scope.go:117] "RemoveContainer" containerID="5e6e5d62d6652e736eaa4845f9cfd9289d74e17fa8c30906a385c6197f322dfb" Jul 13 21:25:48.899221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:48.898963 2561 scope.go:117] "RemoveContainer" containerID="ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1" Jul 13 21:25:48.899221 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:48.899198 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9_odh-ai-gateway-infra(3ffeb5b4-0aed-476d-b569-5c48cb22a0f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" Jul 13 21:25:49.905464 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:25:49.905433 2561 scope.go:117] "RemoveContainer" containerID="ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1" Jul 13 21:25:49.905889 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:25:49.905674 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9_odh-ai-gateway-infra(3ffeb5b4-0aed-476d-b569-5c48cb22a0f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" Jul 13 21:26:03.828838 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:03.828804 2561 scope.go:117] "RemoveContainer" containerID="ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1" Jul 13 21:26:04.983599 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:04.983569 2561 generic.go:356] "Generic (PLEG): container finished" podID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerID="223a29e63010378fcba497c958e11f31477caaf502fb2d7936d0f8ec4bbd52ea" exitCode=0 Jul 13 21:26:04.984026 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:04.983628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerDied","Data":"223a29e63010378fcba497c958e11f31477caaf502fb2d7936d0f8ec4bbd52ea"} Jul 13 21:26:04.984026 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:04.983676 2561 scope.go:117] "RemoveContainer" containerID="ceb9f7cdf1bc2ffaa5d6c21cb52d6b45b5e947bd342aec032acacae70399a7a1" Jul 13 21:26:06.013385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.013352 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:26:06.131416 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.131392 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:26:06.172983 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.172955 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca\") pod \"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4\" (UID: \"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4\") " Jul 13 21:26:06.173289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.173268 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" (UID: "3ffeb5b4-0aed-476d-b569-5c48cb22a0f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:26:06.274442 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.274369 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4-maas-api-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:26:06.996646 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.996611 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" event={"ID":"3ffeb5b4-0aed-476d-b569-5c48cb22a0f4","Type":"ContainerDied","Data":"bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c"} Jul 13 21:26:06.996646 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.996645 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9" Jul 13 21:26:06.996887 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:06.996659 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb48eb5e1e9989146a3e26728d13eeaa237988fb46ff65fa86976ee4effb889c" Jul 13 21:26:07.041594 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:07.041560 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:26:07.053023 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:07.052991 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-mbmr9"] Jul 13 21:26:08.835504 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:08.835461 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" path="/var/lib/kubelet/pods/3ffeb5b4-0aed-476d-b569-5c48cb22a0f4/volumes" Jul 13 21:26:46.117549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117464 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c"] Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117935 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117949 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117968 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117974 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117979 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117985 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2754e399-c028-40d9-97b5-2c7b3d095a34" containerName="istio-proxy" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.117998 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.118002 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.118058 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d650c0-d180-410f-bcd7-58c64e15c648" containerName="istio-proxy" Jul 13 21:26:46.118289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.118067 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.118074 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.118289 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.118081 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ffeb5b4-0aed-476d-b569-5c48cb22a0f4" containerName="revoke-keys" Jul 13 21:26:46.120430 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.120413 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:46.122981 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.122960 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:26:46.128999 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.128929 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c"] Jul 13 21:26:46.236606 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.236577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c\" (UID: \"f24ddb3c-c1d1-4383-beab-ab08e7042c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:46.337785 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.337735 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c\" (UID: \"f24ddb3c-c1d1-4383-beab-ab08e7042c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:46.338368 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.338350 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c\" (UID: \"f24ddb3c-c1d1-4383-beab-ab08e7042c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:46.432621 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.432531 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:46.568051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:46.568029 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c"] Jul 13 21:26:46.571155 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:26:46.571123 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24ddb3c_c1d1_4383_beab_ab08e7042c6f.slice/crio-1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f WatchSource:0}: Error finding container 1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f: Status 404 returned error can't find the container with id 1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f Jul 13 21:26:47.202546 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:47.202498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" event={"ID":"f24ddb3c-c1d1-4383-beab-ab08e7042c6f","Type":"ContainerStarted","Data":"c1be9582f7415c1c8270ecfad980e991dafd315e2f249d7b0e00516079ed869c"} Jul 13 21:26:47.203001 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:47.202581 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" event={"ID":"f24ddb3c-c1d1-4383-beab-ab08e7042c6f","Type":"ContainerStarted","Data":"1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f"} Jul 13 21:26:47.220110 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:47.220062 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" podStartSLOduration=1.220045727 podStartE2EDuration="1.220045727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:26:47.216882255 +0000 UTC m=+1982.945942174" watchObservedRunningTime="2026-07-13 21:26:47.220045727 +0000 UTC m=+1982.949105627" Jul 13 21:26:48.208652 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:48.208614 2561 generic.go:356] "Generic (PLEG): container finished" podID="f24ddb3c-c1d1-4383-beab-ab08e7042c6f" containerID="c1be9582f7415c1c8270ecfad980e991dafd315e2f249d7b0e00516079ed869c" exitCode=0 Jul 13 21:26:48.209069 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:48.208669 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" event={"ID":"f24ddb3c-c1d1-4383-beab-ab08e7042c6f","Type":"ContainerDied","Data":"c1be9582f7415c1c8270ecfad980e991dafd315e2f249d7b0e00516079ed869c"} Jul 13 21:26:49.355038 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:49.355014 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:49.364821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:49.364797 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca\") pod \"f24ddb3c-c1d1-4383-beab-ab08e7042c6f\" (UID: \"f24ddb3c-c1d1-4383-beab-ab08e7042c6f\") " Jul 13 21:26:49.365105 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:49.365085 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f24ddb3c-c1d1-4383-beab-ab08e7042c6f" (UID: "f24ddb3c-c1d1-4383-beab-ab08e7042c6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:26:49.465995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:49.465909 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f24ddb3c-c1d1-4383-beab-ab08e7042c6f-maas-api-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:26:50.220651 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:50.220621 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" Jul 13 21:26:50.220844 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:50.220661 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c" event={"ID":"f24ddb3c-c1d1-4383-beab-ab08e7042c6f","Type":"ContainerDied","Data":"1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f"} Jul 13 21:26:50.220844 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:26:50.220708 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1073e4567f19e3e488155b3ac5c13f1f232a61f7b4dbcd4aae031f1aed24497f" Jul 13 21:27:12.487927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.487878 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:27:12.488525 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.488489 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f24ddb3c-c1d1-4383-beab-ab08e7042c6f" containerName="revoke-keys" Jul 13 21:27:12.488525 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.488512 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f24ddb3c-c1d1-4383-beab-ab08e7042c6f" containerName="revoke-keys" Jul 13 21:27:12.488674 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.488665 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f24ddb3c-c1d1-4383-beab-ab08e7042c6f" containerName="revoke-keys" Jul 13 21:27:12.491089 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.491067 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.493677 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.493657 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f555b378-openshift-default-dockercfg-9mrvp\"" Jul 13 21:27:12.503219 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.503195 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:27:12.557589 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557554 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557805 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557619 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557805 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557805 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557715 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557953 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557829 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557953 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557863 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557953 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.557953 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557942 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.558081 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.557977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7sw\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658445 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658413 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658469 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7sw\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658499 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658534 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658597 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658897 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658897 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.658897 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.658706 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.659085 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.659056 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.659144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.659099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.659191 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.659169 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.659237 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.659221 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.659342 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.659321 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.660994 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.660968 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.661216 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.661198 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.666919 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.666896 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7sw\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.667007 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.666926 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token\") pod \"e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.804957 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.804869 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:12.944948 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.944923 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:27:12.946965 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:27:12.946935 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba8dae_c4bf_4b86_814e_4d171fd3234d.slice/crio-db01de5982e8135e2c18543eb5cf40c2bb7556cd3b465508ab8da21adb161ff8 WatchSource:0}: Error finding container db01de5982e8135e2c18543eb5cf40c2bb7556cd3b465508ab8da21adb161ff8: Status 404 returned error can't find the container with id db01de5982e8135e2c18543eb5cf40c2bb7556cd3b465508ab8da21adb161ff8 Jul 13 21:27:12.949469 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.949442 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:12.949567 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.949516 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:12.949642 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:12.949585 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:13.338915 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:13.338876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerStarted","Data":"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0"} Jul 13 21:27:13.339086 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:13.338939 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerStarted","Data":"db01de5982e8135e2c18543eb5cf40c2bb7556cd3b465508ab8da21adb161ff8"} Jul 13 21:27:13.360121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:13.360070 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podStartSLOduration=1.360054125 podStartE2EDuration="1.360054125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:27:13.358558141 +0000 UTC m=+2009.087618064" watchObservedRunningTime="2026-07-13 21:27:13.360054125 +0000 UTC m=+2009.089114026" Jul 13 21:27:13.805508 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:13.805473 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:14.810623 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:14.810590 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:15.352941 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:15.352903 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:15.353927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:15.353906 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:24.412531 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.412493 2561 generic.go:356] "Generic (PLEG): container finished" podID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerID="4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0" exitCode=0 Jul 13 21:27:24.412951 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.412570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerDied","Data":"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0"} Jul 13 21:27:24.413002 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.412973 2561 scope.go:117] "RemoveContainer" containerID="4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0" Jul 13 21:27:24.413924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.413899 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:24.413999 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.413963 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:24.414041 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:24.413999 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:27:25.419909 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:25.419879 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerStarted","Data":"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9"} Jul 13 21:27:25.806931 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:25.806835 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:25.807121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:25.807005 2561 patch_prober.go:28] interesting pod/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 21:27:25.807200 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:25.807123 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 21:27:26.805330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:26.805295 2561 patch_prober.go:28] interesting pod/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 21:27:26.805934 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:26.805904 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 21:27:27.805396 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:27.805348 2561 patch_prober.go:28] interesting pod/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 21:27:27.805926 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:27.805420 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 21:27:28.805467 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:28.805427 2561 patch_prober.go:28] interesting pod/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 13 21:27:28.806115 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:28.806083 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 13 21:27:29.809128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:29.809100 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:29.809523 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:29.809375 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:27:29.810088 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:27:29.810073 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:28:33.332985 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:33.332954 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:28:33.333514 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:33.333151 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" containerID="cri-o://daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9" gracePeriod=30 Jul 13 21:28:38.600319 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.600296 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:28:38.652770 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652728 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px7sw\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.652922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652781 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.652922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652811 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.652922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652901 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653067 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652929 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653067 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.652958 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653067 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653018 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653067 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653056 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653278 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653102 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert\") pod \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\" (UID: \"9bba8dae-c4bf-4b86-814e-4d171fd3234d\") " Jul 13 21:28:38.653538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653208 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:28:38.653691 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653674 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:28:38.653788 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653303 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:28:38.653788 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.653469 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:28:38.654148 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.654117 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:28:38.655257 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.655190 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw" (OuterVolumeSpecName: "kube-api-access-px7sw") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "kube-api-access-px7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:28:38.655450 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.655417 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:28:38.655787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.655743 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:28:38.655883 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.655833 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9bba8dae-c4bf-4b86-814e-4d171fd3234d" (UID: "9bba8dae-c4bf-4b86-814e-4d171fd3234d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:28:38.753952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753911 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.753952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753947 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.753952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753956 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.753952 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753966 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.754226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753975 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.754226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753984 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.754226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.753993 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px7sw\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-kube-api-access-px7sw\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.754226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.754001 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bba8dae-c4bf-4b86-814e-4d171fd3234d-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.754226 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.754009 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bba8dae-c4bf-4b86-814e-4d171fd3234d-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:28:38.800483 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.800452 2561 generic.go:356] "Generic (PLEG): container finished" podID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerID="daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9" exitCode=0 Jul 13 21:28:38.800654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.800510 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" Jul 13 21:28:38.800654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.800542 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerDied","Data":"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9"} Jul 13 21:28:38.800654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.800605 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h" event={"ID":"9bba8dae-c4bf-4b86-814e-4d171fd3234d","Type":"ContainerDied","Data":"db01de5982e8135e2c18543eb5cf40c2bb7556cd3b465508ab8da21adb161ff8"} Jul 13 21:28:38.800654 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.800640 2561 scope.go:117] "RemoveContainer" containerID="daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9" Jul 13 21:28:38.810584 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.810563 2561 scope.go:117] "RemoveContainer" containerID="4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0" Jul 13 21:28:38.818680 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.818664 2561 scope.go:117] "RemoveContainer" containerID="daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9" Jul 13 21:28:38.818930 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:28:38.818912 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9\": container with ID starting with daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9 not found: ID does not exist" containerID="daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9" Jul 13 21:28:38.818995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.818941 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9"} err="failed to get container status \"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9\": rpc error: code = NotFound desc = could not find container \"daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9\": container with ID starting with daffb798a0ba7f57bce08b1716eb27a806ca20c3a61fbb99edf355d98f6f36e9 not found: ID does not exist" Jul 13 21:28:38.818995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.818958 2561 scope.go:117] "RemoveContainer" containerID="4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0" Jul 13 21:28:38.819154 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:28:38.819138 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0\": container with ID starting with 4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0 not found: ID does not exist" containerID="4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0" Jul 13 21:28:38.819193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.819161 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0"} err="failed to get container status \"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0\": rpc error: code = NotFound desc = could not find container \"4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0\": container with ID starting with 4a9b35c85dbb58e0018374b7c3af87ca193855f94d77d3f9c12b86c58825adf0 not found: ID does not exist" Jul 13 21:28:38.823173 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.823151 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:28:38.826941 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.826921 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f555b378-openshift-default-bf968bf7f-brl5h"] Jul 13 21:28:38.835062 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:38.835041 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" path="/var/lib/kubelet/pods/9bba8dae-c4bf-4b86-814e-4d171fd3234d/volumes" Jul 13 21:28:44.882093 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:44.882065 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:28:44.885669 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:28:44.885643 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:30:09.513218 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.513179 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:09.513916 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.513861 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.513916 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.513881 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.513916 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.513895 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.513916 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.513903 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.514141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.514013 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.514141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.514027 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bba8dae-c4bf-4b86-814e-4d171fd3234d" containerName="istio-proxy" Jul 13 21:30:09.517377 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.517356 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.519989 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.519970 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-543f558f-openshift-default-dockercfg-smgrt\"" Jul 13 21:30:09.527306 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.527282 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:09.554030 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.553999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554090 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554368 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554368 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554324 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554368 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554354 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.554475 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.554374 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf9m\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.654903 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.654865 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.654940 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.654969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.654994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf9m\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655036 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655079 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655070 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655107 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655137 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655330 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655492 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655470 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655747 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655723 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655876 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655817 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.655876 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.655838 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.657420 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.657399 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.657646 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.657626 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.663017 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.662997 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.663154 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.663135 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf9m\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m\") pod \"e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.828364 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.828269 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:09.975026 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.975001 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:09.976945 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:30:09.976909 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f62226_afe7_4974_9d3b_5943ed3a1867.slice/crio-f3df7323b8d2d4fa3bc2956757ed7f575679d780de53404dedbe5fc9978733d2 WatchSource:0}: Error finding container f3df7323b8d2d4fa3bc2956757ed7f575679d780de53404dedbe5fc9978733d2: Status 404 returned error can't find the container with id f3df7323b8d2d4fa3bc2956757ed7f575679d780de53404dedbe5fc9978733d2 Jul 13 21:30:09.979100 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.979071 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:09.979179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.979137 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:09.979179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:09.979164 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:10.288030 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:10.287992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerStarted","Data":"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce"} Jul 13 21:30:10.288186 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:10.288065 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerStarted","Data":"f3df7323b8d2d4fa3bc2956757ed7f575679d780de53404dedbe5fc9978733d2"} Jul 13 21:30:10.308791 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:10.308712 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podStartSLOduration=1.308693828 podStartE2EDuration="1.308693828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:30:10.307997309 +0000 UTC m=+2186.037057233" watchObservedRunningTime="2026-07-13 21:30:10.308693828 +0000 UTC m=+2186.037753728" Jul 13 21:30:10.828781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:10.828738 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:11.833334 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:11.833302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:12.300058 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:12.300019 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:12.300990 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:12.300974 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:26.377041 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.377006 2561 generic.go:356] "Generic (PLEG): container finished" podID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerID="2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce" exitCode=0 Jul 13 21:30:26.377588 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.377082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerDied","Data":"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce"} Jul 13 21:30:26.377588 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.377568 2561 scope.go:117] "RemoveContainer" containerID="2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce" Jul 13 21:30:26.378602 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.378577 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:26.378664 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.378640 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:26.378702 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:26.378672 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:30:27.300681 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:27.300650 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:27.386657 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:27.386619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerStarted","Data":"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931"} Jul 13 21:30:27.828735 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:27.828695 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:27.830047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:27.830021 2561 patch_prober.go:28] interesting pod/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 13 21:30:27.830164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:27.830071 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 13 21:30:28.829170 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:28.829132 2561 patch_prober.go:28] interesting pod/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 13 21:30:28.829596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:28.829192 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 13 21:30:29.829121 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:29.829085 2561 patch_prober.go:28] interesting pod/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 13 21:30:29.829297 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:29.829155 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 13 21:30:30.828893 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:30.828847 2561 patch_prober.go:28] interesting pod/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 13 21:30:30.829063 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:30.828931 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 13 21:30:31.833102 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:31.833065 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:31.833502 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:31.833327 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:31.834049 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:31.834027 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:44.985500 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:44.985468 2561 scope.go:117] "RemoveContainer" containerID="3f640f0326df4adc65c0b85a71f814f4278ceb77a9a38ae51f46bb58ee07e4e7" Jul 13 21:30:49.520388 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:49.520350 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:49.520873 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:49.520579 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" containerID="cri-o://2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931" gracePeriod=30 Jul 13 21:30:54.776251 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.776224 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:54.865778 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865721 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865798 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865841 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865884 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865931 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xf9m\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.865992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.865968 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.866305 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866011 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.866305 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data\") pod \"90f62226-afe7-4974-9d3b-5943ed3a1867\" (UID: \"90f62226-afe7-4974-9d3b-5943ed3a1867\") " Jul 13 21:30:54.866305 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866068 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:30:54.866305 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866290 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:30:54.866528 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866312 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:30:54.866528 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866352 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:30:54.866973 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866944 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data" (OuterVolumeSpecName: "istio-data") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:30:54.866973 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.866952 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f62226-afe7-4974-9d3b-5943ed3a1867-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.867923 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.867847 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.867923 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.867877 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.867923 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.867897 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.873906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.868712 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m" (OuterVolumeSpecName: "kube-api-access-5xf9m") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "kube-api-access-5xf9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:30:54.873906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.869591 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:30:54.873906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.869849 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token" (OuterVolumeSpecName: "istio-token") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:30:54.876631 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.876604 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90f62226-afe7-4974-9d3b-5943ed3a1867" (UID: "90f62226-afe7-4974-9d3b-5943ed3a1867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:30:54.968325 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.968288 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.968325 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.968318 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.968541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.968350 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xf9m\" (UniqueName: \"kubernetes.io/projected/90f62226-afe7-4974-9d3b-5943ed3a1867-kube-api-access-5xf9m\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.968541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.968360 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:54.968541 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:54.968369 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f62226-afe7-4974-9d3b-5943ed3a1867-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:30:55.550558 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.550518 2561 generic.go:356] "Generic (PLEG): container finished" podID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerID="2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931" exitCode=0 Jul 13 21:30:55.550729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.550599 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerDied","Data":"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931"} Jul 13 21:30:55.550729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.550615 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" Jul 13 21:30:55.550729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.550692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn" event={"ID":"90f62226-afe7-4974-9d3b-5943ed3a1867","Type":"ContainerDied","Data":"f3df7323b8d2d4fa3bc2956757ed7f575679d780de53404dedbe5fc9978733d2"} Jul 13 21:30:55.550729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.550716 2561 scope.go:117] "RemoveContainer" containerID="2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931" Jul 13 21:30:55.563379 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.563348 2561 scope.go:117] "RemoveContainer" containerID="2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce" Jul 13 21:30:55.574047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.574025 2561 scope.go:117] "RemoveContainer" containerID="2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931" Jul 13 21:30:55.574392 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:30:55.574345 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931\": container with ID starting with 2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931 not found: ID does not exist" containerID="2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931" Jul 13 21:30:55.574392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.574375 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931"} err="failed to get container status \"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931\": rpc error: code = NotFound desc = could not find container \"2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931\": container with ID starting with 2aaad7faa7cb038d64ef8cd47d53226cdd76ac120cc727c2706c630329a9a931 not found: ID does not exist" Jul 13 21:30:55.574392 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.574394 2561 scope.go:117] "RemoveContainer" containerID="2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce" Jul 13 21:30:55.574686 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:30:55.574669 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce\": container with ID starting with 2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce not found: ID does not exist" containerID="2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce" Jul 13 21:30:55.574740 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.574692 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce"} err="failed to get container status \"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce\": rpc error: code = NotFound desc = could not find container \"2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce\": container with ID starting with 2eae233f977edfa66de6ce8b1e19cf4651cdbf7d97a0f11ecfb43157192165ce not found: ID does not exist" Jul 13 21:30:55.576071 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.576051 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:55.580629 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:55.580604 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-543f558f-openshift-default-964cdd66c-6zmgn"] Jul 13 21:30:56.834927 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:30:56.834892 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" path="/var/lib/kubelet/pods/90f62226-afe7-4974-9d3b-5943ed3a1867/volumes" Jul 13 21:31:09.254415 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254375 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:31:09.254924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254820 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:31:09.254924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254832 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:31:09.254924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254843 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:31:09.254924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254849 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:31:09.254924 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.254907 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:31:09.259517 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.259497 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.262746 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.262722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-11bd6c7a-openshift-default-dockercfg-mgsx4\"" Jul 13 21:31:09.273906 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.273884 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.299853 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.299959 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882kh\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300007 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300113 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300175 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300202 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.300693 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.300245 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401017 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.400988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401017 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401021 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401261 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401261 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401261 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401132 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882kh\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401485 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401446 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401485 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401701 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401832 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401808 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401929 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401908 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.401991 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401954 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.402047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.401965 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.402047 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.402005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.402307 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.402282 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.403991 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.403969 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.404295 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.404278 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.409516 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.409492 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.409914 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.409896 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-882kh\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh\") pod \"e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.570811 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.570691 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:09.716712 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.716690 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:31:09.718917 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:31:09.718882 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f0d98f_e9f4_438c_bdaf_3ab4a87470d2.slice/crio-999e820aee24cd2d17092179e39591ee923678b0b9059ecab98c028e345a1fbc WatchSource:0}: Error finding container 999e820aee24cd2d17092179e39591ee923678b0b9059ecab98c028e345a1fbc: Status 404 returned error can't find the container with id 999e820aee24cd2d17092179e39591ee923678b0b9059ecab98c028e345a1fbc Jul 13 21:31:09.721669 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.721597 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:09.721798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.721698 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:09.721798 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:09.721737 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:10.631433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:10.631397 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerStarted","Data":"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5"} Jul 13 21:31:10.631835 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:10.631452 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerStarted","Data":"999e820aee24cd2d17092179e39591ee923678b0b9059ecab98c028e345a1fbc"} Jul 13 21:31:10.656011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:10.655963 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podStartSLOduration=1.655946234 podStartE2EDuration="1.655946234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:31:10.65387434 +0000 UTC m=+2246.382934240" watchObservedRunningTime="2026-07-13 21:31:10.655946234 +0000 UTC m=+2246.385006133" Jul 13 21:31:11.571515 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:11.571479 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:11.576165 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:11.576138 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:11.637190 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:11.637164 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:11.638211 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:11.638190 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:26.819871 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:26.819442 2561 patch_prober.go:28] interesting pod/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:33296->10.132.0.62:15021: read: connection reset by peer" start-of-body= Jul 13 21:31:26.819871 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:26.819523 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:33296->10.132.0.62:15021: read: connection reset by peer" Jul 13 21:31:27.730586 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.730552 2561 generic.go:356] "Generic (PLEG): container finished" podID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerID="ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5" exitCode=0 Jul 13 21:31:27.730810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.730636 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerDied","Data":"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5"} Jul 13 21:31:27.731122 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.731103 2561 scope.go:117] "RemoveContainer" containerID="ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5" Jul 13 21:31:27.732228 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.732202 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:27.732316 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.732267 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:27.732316 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:27.732298 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:31:28.739587 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:28.739551 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerStarted","Data":"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff"} Jul 13 21:31:29.570983 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:29.570952 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:29.571259 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:29.571227 2561 patch_prober.go:28] interesting pod/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 21:31:29.571379 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:29.571279 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 21:31:30.571258 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:30.571217 2561 patch_prober.go:28] interesting pod/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 21:31:30.571640 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:30.571276 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 21:31:31.571124 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:31.571084 2561 patch_prober.go:28] interesting pod/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 21:31:31.571328 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:31.571146 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 21:31:32.571199 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:32.571164 2561 patch_prober.go:28] interesting pod/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 13 21:31:32.571385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:32.571215 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 13 21:31:33.575429 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:33.575395 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:33.575922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:33.575681 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:31:33.576461 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:31:33.576445 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:32:45.018436 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:32:45.018350 2561 scope.go:117] "RemoveContainer" containerID="223a29e63010378fcba497c958e11f31477caaf502fb2d7936d0f8ec4bbd52ea" Jul 13 21:33:22.646003 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:22.645971 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:33:22.646479 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:22.646203 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" containerID="cri-o://3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff" gracePeriod=30 Jul 13 21:33:27.911643 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.911573 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:33:27.975920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.975873 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.975920 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.975928 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.975958 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976000 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976048 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976072 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976107 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976200 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:33:27.976444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976287 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:33:27.976555 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976524 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:33:27.976614 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976564 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-882kh\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.976667 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976627 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs\") pod \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\" (UID: \"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2\") " Jul 13 21:33:27.977000 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976971 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:33:27.977000 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976974 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:27.977193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.977020 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:27.977193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.977038 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:27.977193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.976993 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:33:27.978390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.978358 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:33:27.978500 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.978406 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:33:27.978639 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.978615 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:33:27.978810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:27.978790 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh" (OuterVolumeSpecName: "kube-api-access-882kh") pod "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" (UID: "31f0d98f-e9f4-438c-bdaf-3ab4a87470d2"). InnerVolumeSpecName "kube-api-access-882kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:33:28.078474 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078438 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-882kh\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-kube-api-access-882kh\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.078474 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078468 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.078474 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078477 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.078474 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078486 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.078739 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078493 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.078739 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.078501 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:33:28.370049 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.370012 2561 generic.go:356] "Generic (PLEG): container finished" podID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerID="3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff" exitCode=0 Jul 13 21:33:28.370227 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.370060 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" Jul 13 21:33:28.370227 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.370093 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerDied","Data":"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff"} Jul 13 21:33:28.370227 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.370154 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj" event={"ID":"31f0d98f-e9f4-438c-bdaf-3ab4a87470d2","Type":"ContainerDied","Data":"999e820aee24cd2d17092179e39591ee923678b0b9059ecab98c028e345a1fbc"} Jul 13 21:33:28.370227 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.370182 2561 scope.go:117] "RemoveContainer" containerID="3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff" Jul 13 21:33:28.380398 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.380380 2561 scope.go:117] "RemoveContainer" containerID="ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5" Jul 13 21:33:28.389888 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.389864 2561 scope.go:117] "RemoveContainer" containerID="3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff" Jul 13 21:33:28.390159 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:33:28.390139 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff\": container with ID starting with 3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff not found: ID does not exist" containerID="3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff" Jul 13 21:33:28.390221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.390168 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff"} err="failed to get container status \"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff\": rpc error: code = NotFound desc = could not find container \"3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff\": container with ID starting with 3cde453119c2727077d9077a3e1793d9319862ce293f81add752809a1e41eaff not found: ID does not exist" Jul 13 21:33:28.390221 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.390186 2561 scope.go:117] "RemoveContainer" containerID="ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5" Jul 13 21:33:28.390436 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:33:28.390419 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5\": container with ID starting with ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5 not found: ID does not exist" containerID="ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5" Jul 13 21:33:28.390475 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.390442 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5"} err="failed to get container status \"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5\": rpc error: code = NotFound desc = could not find container \"ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5\": container with ID starting with ad85e15b6a85e3330984f9625b69ef775d3863008582d0aa2b00f47dca64e4a5 not found: ID does not exist" Jul 13 21:33:28.394504 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.394480 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:33:28.398251 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.398231 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-11bd6c7a-openshift-default-5854c6c66c-7hcwj"] Jul 13 21:33:28.835385 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:28.835290 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" path="/var/lib/kubelet/pods/31f0d98f-e9f4-438c-bdaf-3ab4a87470d2/volumes" Jul 13 21:33:44.912635 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:44.912603 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:33:44.920110 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:33:44.920089 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:34:17.320976 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.320938 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:34:17.321473 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321420 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.321473 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321436 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.321473 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321456 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.321473 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321462 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.321628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321525 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f62226-afe7-4974-9d3b-5943ed3a1867" containerName="istio-proxy" Jul 13 21:34:17.321628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321535 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.321628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.321541 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f0d98f-e9f4-438c-bdaf-3ab4a87470d2" containerName="istio-proxy" Jul 13 21:34:17.324701 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.324679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.327237 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.327212 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-613d27-openshift-default-dockercfg-mglmh\"" Jul 13 21:34:17.338656 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.338635 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:34:17.426036 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426001 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426052 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426115 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426247 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426298 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426326 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.426394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.426347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnx4\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527382 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527347 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527583 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527583 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527445 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527583 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527486 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527907 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.527907 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.527798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528125 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528072 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528125 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528125 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528110 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnx4\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528281 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528129 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528404 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528360 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528508 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528401 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.528554 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.528510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.530472 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.530445 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.530891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.530872 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.535155 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.535128 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.536680 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.536658 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnx4\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4\") pod \"e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.635417 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.635390 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:17.782261 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.782185 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:34:17.788499 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:34:17.788456 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc997c983_0e64_443d_b0a0_5eddab1cf879.slice/crio-635c184130658811b0f281e9069e093bbede67d3050924024593ed87177df56b WatchSource:0}: Error finding container 635c184130658811b0f281e9069e093bbede67d3050924024593ed87177df56b: Status 404 returned error can't find the container with id 635c184130658811b0f281e9069e093bbede67d3050924024593ed87177df56b Jul 13 21:34:17.789018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.788980 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:17.789131 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.789067 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:17.789131 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:17.789104 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:18.639947 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:18.639905 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerStarted","Data":"ad864e69ffb3867fe9d1456a4f13832ea90cae2018f9761adfd008de9d906d35"} Jul 13 21:34:18.640421 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:18.639962 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerStarted","Data":"635c184130658811b0f281e9069e093bbede67d3050924024593ed87177df56b"} Jul 13 21:34:18.661765 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:18.661705 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podStartSLOduration=1.661691367 podStartE2EDuration="1.661691367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:34:18.659495386 +0000 UTC m=+2434.388555307" watchObservedRunningTime="2026-07-13 21:34:18.661691367 +0000 UTC m=+2434.390751266" Jul 13 21:34:19.635608 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:19.635574 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:19.640639 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:19.640611 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:19.645266 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:19.645244 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:19.646035 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:19.646015 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:35.687018 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:35.686980 2561 patch_prober.go:28] interesting pod/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 21:34:35.687533 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:35.687061 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 21:34:37.754733 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.754697 2561 generic.go:356] "Generic (PLEG): container finished" podID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerID="ad864e69ffb3867fe9d1456a4f13832ea90cae2018f9761adfd008de9d906d35" exitCode=0 Jul 13 21:34:37.755167 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.754788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerDied","Data":"ad864e69ffb3867fe9d1456a4f13832ea90cae2018f9761adfd008de9d906d35"} Jul 13 21:34:37.755215 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.755187 2561 scope.go:117] "RemoveContainer" containerID="ad864e69ffb3867fe9d1456a4f13832ea90cae2018f9761adfd008de9d906d35" Jul 13 21:34:37.756168 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.756140 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:37.756227 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.756210 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:37.756264 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:37.756243 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:34:38.761882 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:38.761848 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerStarted","Data":"6531cb560af150a841da8f2c82acf86a61940ae19f537e63d658c3ae23a24afc"} Jul 13 21:34:39.635951 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:39.635911 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:39.636149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:39.636082 2561 patch_prober.go:28] interesting pod/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 13 21:34:39.636149 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:39.636128 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 13 21:34:40.636082 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:40.636046 2561 patch_prober.go:28] interesting pod/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 13 21:34:40.636507 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:40.636105 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 13 21:34:41.636318 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:41.636281 2561 patch_prober.go:28] interesting pod/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 13 21:34:41.636694 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:41.636335 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 13 21:34:42.636250 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:42.636202 2561 patch_prober.go:28] interesting pod/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 13 21:34:42.636472 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:42.636263 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 13 21:34:43.639373 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:43.639342 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:43.639803 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:43.639676 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:34:43.640343 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:34:43.640323 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:35:33.283638 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.283596 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2"] Jul 13 21:35:33.287846 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.287817 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:33.290509 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.290488 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:35:33.295501 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.295479 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2"] Jul 13 21:35:33.418092 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.418061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2\" (UID: \"8d12fa30-f5ba-4f42-bb1c-c04f7affed31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:33.519423 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.519385 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2\" (UID: \"8d12fa30-f5ba-4f42-bb1c-c04f7affed31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:33.520039 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.520021 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2\" (UID: \"8d12fa30-f5ba-4f42-bb1c-c04f7affed31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:33.600812 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.600779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:33.949524 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:33.949493 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2"] Jul 13 21:35:33.951452 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:35:33.951424 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d12fa30_f5ba_4f42_bb1c_c04f7affed31.slice/crio-2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a WatchSource:0}: Error finding container 2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a: Status 404 returned error can't find the container with id 2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a Jul 13 21:35:34.059504 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:34.059462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" event={"ID":"8d12fa30-f5ba-4f42-bb1c-c04f7affed31","Type":"ContainerStarted","Data":"2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a"} Jul 13 21:35:35.067078 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:35.066986 2561 generic.go:356] "Generic (PLEG): container finished" podID="8d12fa30-f5ba-4f42-bb1c-c04f7affed31" containerID="42a7c53b54a2d32a49d3e11732cdd370e95c11c388d9ba45a885769e5cd32893" exitCode=0 Jul 13 21:35:35.067078 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:35.067038 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" event={"ID":"8d12fa30-f5ba-4f42-bb1c-c04f7affed31","Type":"ContainerDied","Data":"42a7c53b54a2d32a49d3e11732cdd370e95c11c388d9ba45a885769e5cd32893"} Jul 13 21:35:36.212082 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:36.212057 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:36.347719 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:36.347694 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca\") pod \"8d12fa30-f5ba-4f42-bb1c-c04f7affed31\" (UID: \"8d12fa30-f5ba-4f42-bb1c-c04f7affed31\") " Jul 13 21:35:36.348062 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:36.348038 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d12fa30-f5ba-4f42-bb1c-c04f7affed31" (UID: "8d12fa30-f5ba-4f42-bb1c-c04f7affed31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:35:36.448999 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:36.448963 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d12fa30-f5ba-4f42-bb1c-c04f7affed31-maas-api-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:35:37.077928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:37.077854 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" Jul 13 21:35:37.077928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:37.077890 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2" event={"ID":"8d12fa30-f5ba-4f42-bb1c-c04f7affed31","Type":"ContainerDied","Data":"2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a"} Jul 13 21:35:37.078141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:37.077935 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea72c2934bc9c4bad2a5d86c3840c0b783ed67486636e5cc60805f6fe10b22a" Jul 13 21:35:59.048512 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:59.048469 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:35:59.050000 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:35:59.049955 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" containerID="cri-o://6531cb560af150a841da8f2c82acf86a61940ae19f537e63d658c3ae23a24afc" gracePeriod=30 Jul 13 21:36:04.222023 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.221988 2561 generic.go:356] "Generic (PLEG): container finished" podID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerID="6531cb560af150a841da8f2c82acf86a61940ae19f537e63d658c3ae23a24afc" exitCode=0 Jul 13 21:36:04.222441 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.222062 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerDied","Data":"6531cb560af150a841da8f2c82acf86a61940ae19f537e63d658c3ae23a24afc"} Jul 13 21:36:04.222441 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.222125 2561 scope.go:117] "RemoveContainer" containerID="ad864e69ffb3867fe9d1456a4f13832ea90cae2018f9761adfd008de9d906d35" Jul 13 21:36:04.318622 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.318599 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:36:04.405071 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405038 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405099 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405126 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbnx4\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405171 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405213 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405259 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405396 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405458 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405482 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data\") pod \"c997c983-0e64-443d-b0a0-5eddab1cf879\" (UID: \"c997c983-0e64-443d-b0a0-5eddab1cf879\") " Jul 13 21:36:04.405626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405482 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:36:04.405626 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405545 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:36:04.405977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405681 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:36:04.405977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405898 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.405977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405903 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:36:04.405977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405920 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.405977 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.405934 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.406724 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.406679 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data" (OuterVolumeSpecName: "istio-data") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:36:04.407376 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.407348 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4" (OuterVolumeSpecName: "kube-api-access-zbnx4") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "kube-api-access-zbnx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:36:04.407464 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.407392 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:36:04.407464 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.407422 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token" (OuterVolumeSpecName: "istio-token") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:36:04.407584 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.407567 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c997c983-0e64-443d-b0a0-5eddab1cf879" (UID: "c997c983-0e64-443d-b0a0-5eddab1cf879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:36:04.506692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506661 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.506692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506689 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnx4\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-kube-api-access-zbnx4\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.506918 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506702 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.506918 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506715 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.506918 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506726 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c997c983-0e64-443d-b0a0-5eddab1cf879-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:04.506918 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:04.506736 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c997c983-0e64-443d-b0a0-5eddab1cf879-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:36:05.228656 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:05.228616 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" event={"ID":"c997c983-0e64-443d-b0a0-5eddab1cf879","Type":"ContainerDied","Data":"635c184130658811b0f281e9069e093bbede67d3050924024593ed87177df56b"} Jul 13 21:36:05.229113 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:05.228668 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c" Jul 13 21:36:05.229113 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:05.228722 2561 scope.go:117] "RemoveContainer" containerID="6531cb560af150a841da8f2c82acf86a61940ae19f537e63d658c3ae23a24afc" Jul 13 21:36:05.250034 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:05.250009 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:36:05.252825 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:05.252803 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-613d27-openshift-default-65fdb9f967-chs6c"] Jul 13 21:36:06.834544 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:06.834501 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" path="/var/lib/kubelet/pods/c997c983-0e64-443d-b0a0-5eddab1cf879/volumes" Jul 13 21:36:26.081196 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081163 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:36:26.081635 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081616 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:26.081821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081637 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:26.081821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081696 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d12fa30-f5ba-4f42-bb1c-c04f7affed31" containerName="revoke-keys" Jul 13 21:36:26.081821 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081706 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d12fa30-f5ba-4f42-bb1c-c04f7affed31" containerName="revoke-keys" Jul 13 21:36:26.081995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081828 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d12fa30-f5ba-4f42-bb1c-c04f7affed31" containerName="revoke-keys" Jul 13 21:36:26.081995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081846 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:26.081995 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.081856 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:26.086002 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.085979 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.088741 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.088718 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eb35a4-openshift-default-dockercfg-8bgcb\"" Jul 13 21:36:26.099011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.098983 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:36:26.205956 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.205925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.205977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vzk\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.205999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206180 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206204 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206231 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.206365 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.206281 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307566 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307527 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307587 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vzk\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307607 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307636 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307681 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.307797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307721 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308189 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.307792 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308189 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.308136 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308409 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.308187 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308409 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.308335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308409 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.308370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.308796 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.308772 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.309988 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.309966 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.310275 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.310253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.316281 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.316253 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.316387 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.316330 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vzk\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk\") pod \"e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.397440 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.397407 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:26.537987 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.537956 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:36:26.540034 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:36:26.540001 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbeef55_d0fb_4d79_8fc5_c0b98576b080.slice/crio-6e40a38f9f3f0fb5baac267aadce6829049513da3784b055f055d72fafe956c0 WatchSource:0}: Error finding container 6e40a38f9f3f0fb5baac267aadce6829049513da3784b055f055d72fafe956c0: Status 404 returned error can't find the container with id 6e40a38f9f3f0fb5baac267aadce6829049513da3784b055f055d72fafe956c0 Jul 13 21:36:26.542182 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.542149 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:26.542291 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.542219 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:26.542291 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:26.542249 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:27.359479 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.359447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerStarted","Data":"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe"} Jul 13 21:36:27.359935 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.359505 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerStarted","Data":"6e40a38f9f3f0fb5baac267aadce6829049513da3784b055f055d72fafe956c0"} Jul 13 21:36:27.381151 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.381101 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podStartSLOduration=1.3810811379999999 podStartE2EDuration="1.381081138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:36:27.379848567 +0000 UTC m=+2563.108908487" watchObservedRunningTime="2026-07-13 21:36:27.381081138 +0000 UTC m=+2563.110141039" Jul 13 21:36:27.397835 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.397802 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:27.399316 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.399291 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 13 21:36:27.399438 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:27.399343 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 13 21:36:28.401525 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:28.401496 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:29.369797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:29.369741 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:29.370728 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:29.370711 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:34.546014 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.545981 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:36:34.546403 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.546392 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:34.546449 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.546405 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c997c983-0e64-443d-b0a0-5eddab1cf879" containerName="istio-proxy" Jul 13 21:36:34.549703 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.549683 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.552223 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.552195 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c6c469-openshift-default-dockercfg-6kbsk\"" Jul 13 21:36:34.563900 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.563878 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:36:34.581510 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581488 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95fw\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581607 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581527 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581607 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581592 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581631 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581692 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581728 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581850 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581822 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581904 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581846 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.581904 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.581879 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682429 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682446 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682540 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682584 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95fw\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682649 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682618 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.682980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.682915 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.683558 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.683512 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.683691 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.683637 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.683835 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.683820 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.684002 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.683979 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.685633 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.685606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.686122 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.686103 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.691840 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.691815 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95fw\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.692110 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.692093 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token\") pod \"e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:34.860662 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:34.860633 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:35.004800 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.004747 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:36:35.007708 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:36:35.007677 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207a1629_ffc7_40c8_b687_c3f7916710e0.slice/crio-02b41ba76e18a0966dc9ed907f8a0ff6e28c9043abe3fb5042ca51fe4b6913af WatchSource:0}: Error finding container 02b41ba76e18a0966dc9ed907f8a0ff6e28c9043abe3fb5042ca51fe4b6913af: Status 404 returned error can't find the container with id 02b41ba76e18a0966dc9ed907f8a0ff6e28c9043abe3fb5042ca51fe4b6913af Jul 13 21:36:35.010468 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.010426 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:35.010574 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.010515 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:35.010574 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.010556 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:35.404912 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.404866 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerStarted","Data":"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549"} Jul 13 21:36:35.405223 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.404964 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerStarted","Data":"02b41ba76e18a0966dc9ed907f8a0ff6e28c9043abe3fb5042ca51fe4b6913af"} Jul 13 21:36:35.428225 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.428177 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podStartSLOduration=1.428163842 podStartE2EDuration="1.428163842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:36:35.426875558 +0000 UTC m=+2571.155935458" watchObservedRunningTime="2026-07-13 21:36:35.428163842 +0000 UTC m=+2571.157223742" Jul 13 21:36:35.861410 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:35.861381 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:36.866025 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:36.865995 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:37.418470 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:37.418422 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:37.419417 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:37.419393 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:43.453416 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.453384 2561 generic.go:356] "Generic (PLEG): container finished" podID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerID="5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe" exitCode=0 Jul 13 21:36:43.453882 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.453455 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerDied","Data":"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe"} Jul 13 21:36:43.453882 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.453870 2561 scope.go:117] "RemoveContainer" containerID="5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe" Jul 13 21:36:43.454987 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.454955 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:43.455111 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.455042 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:43.455177 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:43.455115 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:44.370429 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:44.370394 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:44.461011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:44.460976 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerStarted","Data":"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8"} Jul 13 21:36:45.398830 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:45.398789 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:45.399056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:45.398924 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 13 21:36:45.399056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:45.399007 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 13 21:36:46.398789 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:46.398723 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 13 21:36:46.399383 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:46.399348 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 13 21:36:47.398606 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:47.398571 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 13 21:36:47.398921 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:47.398875 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 13 21:36:48.398872 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:48.398807 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 13 21:36:48.399807 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:48.399122 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 13 21:36:49.402340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:49.402302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:49.403552 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:49.403528 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:49.410850 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:49.410824 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:36:50.504930 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.504898 2561 generic.go:356] "Generic (PLEG): container finished" podID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerID="df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549" exitCode=0 Jul 13 21:36:50.505406 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.504969 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerDied","Data":"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549"} Jul 13 21:36:50.505533 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.505514 2561 scope.go:117] "RemoveContainer" containerID="df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549" Jul 13 21:36:50.506582 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.506554 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:50.506657 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.506640 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:50.506708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:50.506692 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:36:51.515145 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:51.515112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerStarted","Data":"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a"} Jul 13 21:36:51.861831 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:51.861794 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:51.863048 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:51.863025 2561 patch_prober.go:28] interesting pod/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 21:36:51.863155 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:51.863070 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 21:36:52.419179 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:52.419141 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:52.861548 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:52.861520 2561 patch_prober.go:28] interesting pod/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 21:36:52.861975 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:52.861570 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 21:36:53.861178 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:53.861140 2561 patch_prober.go:28] interesting pod/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 21:36:53.861350 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:53.861198 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 21:36:54.861687 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:54.861630 2561 patch_prober.go:28] interesting pod/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 13 21:36:54.862109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:54.861733 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 13 21:36:55.865730 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:55.865701 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:36:55.866562 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:36:55.866542 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:37:15.623314 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:15.623277 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:37:15.624170 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:15.624133 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" containerID="cri-o://9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8" gracePeriod=30 Jul 13 21:37:19.405857 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:19.405820 2561 patch_prober.go:28] interesting pod/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 21:37:19.406238 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:19.405887 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 21:37:20.890575 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.890507 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:37:20.942577 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942548 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942587 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942609 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942626 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942681 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942699 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.942781 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942739 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.943141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942859 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74vzk\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.943141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942904 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:20.943141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.942911 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token\") pod \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\" (UID: \"ebbeef55-d0fb-4d79-8fc5-c0b98576b080\") " Jul 13 21:37:20.943141 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943038 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:20.943361 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943301 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:20.943361 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943325 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:20.943507 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943477 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:20.943572 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943485 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:37:20.943631 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.943581 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:20.945196 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.945169 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk" (OuterVolumeSpecName: "kube-api-access-74vzk") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "kube-api-access-74vzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:37:20.945317 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.945195 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:37:20.945390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.945343 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:20.945390 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:20.945350 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebbeef55-d0fb-4d79-8fc5-c0b98576b080" (UID: "ebbeef55-d0fb-4d79-8fc5-c0b98576b080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:37:21.044747 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044713 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vzk\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-kube-api-access-74vzk\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044747 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044743 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044747 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044770 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044780 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044787 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044797 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.044993 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.044807 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebbeef55-d0fb-4d79-8fc5-c0b98576b080-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:21.679741 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.679702 2561 generic.go:356] "Generic (PLEG): container finished" podID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerID="9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8" exitCode=0 Jul 13 21:37:21.679930 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.679774 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" Jul 13 21:37:21.679930 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.679789 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerDied","Data":"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8"} Jul 13 21:37:21.679930 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.679851 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv" event={"ID":"ebbeef55-d0fb-4d79-8fc5-c0b98576b080","Type":"ContainerDied","Data":"6e40a38f9f3f0fb5baac267aadce6829049513da3784b055f055d72fafe956c0"} Jul 13 21:37:21.679930 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.679876 2561 scope.go:117] "RemoveContainer" containerID="9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8" Jul 13 21:37:21.689152 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.689138 2561 scope.go:117] "RemoveContainer" containerID="5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe" Jul 13 21:37:21.697456 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.697437 2561 scope.go:117] "RemoveContainer" containerID="9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8" Jul 13 21:37:21.697720 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:37:21.697702 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8\": container with ID starting with 9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8 not found: ID does not exist" containerID="9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8" Jul 13 21:37:21.697802 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.697728 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8"} err="failed to get container status \"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8\": rpc error: code = NotFound desc = could not find container \"9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8\": container with ID starting with 9c64ed0c6a49f5ef63df36c113cb6bda9074a2f774434438c130fc902b2904d8 not found: ID does not exist" Jul 13 21:37:21.697802 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.697744 2561 scope.go:117] "RemoveContainer" containerID="5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe" Jul 13 21:37:21.698013 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:37:21.697996 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe\": container with ID starting with 5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe not found: ID does not exist" containerID="5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe" Jul 13 21:37:21.698054 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.698019 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe"} err="failed to get container status \"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe\": rpc error: code = NotFound desc = could not find container \"5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe\": container with ID starting with 5b1b292e05857fc76fe01937396d59eade36d3e935eea7c589ebc413a793cffe not found: ID does not exist" Jul 13 21:37:21.702381 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.702360 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:37:21.706176 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:21.706154 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eb35a4-openshift-default-6df98f567f-426fv"] Jul 13 21:37:22.834879 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:22.834849 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" path="/var/lib/kubelet/pods/ebbeef55-d0fb-4d79-8fc5-c0b98576b080/volumes" Jul 13 21:37:41.726972 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:41.726937 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:37:41.727444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:41.727221 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" containerID="cri-o://a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a" gracePeriod=30 Jul 13 21:37:46.999500 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:46.999426 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:37:47.079570 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079540 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.079792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079600 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.079792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079633 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.079792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079673 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c95fw\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.079792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079709 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.079792 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079736 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.080108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079846 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.080108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079881 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:47.080108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079910 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.080108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.079937 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo\") pod \"207a1629-ffc7-40c8-b687-c3f7916710e0\" (UID: \"207a1629-ffc7-40c8-b687-c3f7916710e0\") " Jul 13 21:37:47.080108 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080055 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:47.080375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080209 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:47.080375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080228 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.080375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080251 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.080540 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080439 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:47.080797 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.080701 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:37:47.082898 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.082838 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw" (OuterVolumeSpecName: "kube-api-access-c95fw") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "kube-api-access-c95fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:37:47.083265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.083224 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:37:47.083265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.083253 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:37:47.086154 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.086120 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "207a1629-ffc7-40c8-b687-c3f7916710e0" (UID: "207a1629-ffc7-40c8-b687-c3f7916710e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:37:47.181292 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181259 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95fw\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-kube-api-access-c95fw\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181292 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181287 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181292 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181298 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181305 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207a1629-ffc7-40c8-b687-c3f7916710e0-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181313 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181321 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.181538 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.181329 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207a1629-ffc7-40c8-b687-c3f7916710e0-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:37:47.824363 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.824323 2561 generic.go:356] "Generic (PLEG): container finished" podID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerID="a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a" exitCode=0 Jul 13 21:37:47.824576 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.824407 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" Jul 13 21:37:47.824576 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.824406 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerDied","Data":"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a"} Jul 13 21:37:47.824704 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.824575 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw" event={"ID":"207a1629-ffc7-40c8-b687-c3f7916710e0","Type":"ContainerDied","Data":"02b41ba76e18a0966dc9ed907f8a0ff6e28c9043abe3fb5042ca51fe4b6913af"} Jul 13 21:37:47.824704 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.824612 2561 scope.go:117] "RemoveContainer" containerID="a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a" Jul 13 21:37:47.835437 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.835416 2561 scope.go:117] "RemoveContainer" containerID="df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549" Jul 13 21:37:47.844185 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.844160 2561 scope.go:117] "RemoveContainer" containerID="a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a" Jul 13 21:37:47.844623 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:37:47.844605 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a\": container with ID starting with a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a not found: ID does not exist" containerID="a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a" Jul 13 21:37:47.844684 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.844633 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a"} err="failed to get container status \"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a\": rpc error: code = NotFound desc = could not find container \"a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a\": container with ID starting with a22a379108f6f6ebcc66412971dc5ca0bf81373f0f4b6c58a68a3f6873d7a66a not found: ID does not exist" Jul 13 21:37:47.844684 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.844651 2561 scope.go:117] "RemoveContainer" containerID="df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549" Jul 13 21:37:47.844965 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:37:47.844931 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549\": container with ID starting with df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549 not found: ID does not exist" containerID="df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549" Jul 13 21:37:47.845068 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.844969 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549"} err="failed to get container status \"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549\": rpc error: code = NotFound desc = could not find container \"df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549\": container with ID starting with df2878d7cd56f8023393a334e073c0d167bc5b1026028e138284f61b80062549 not found: ID does not exist" Jul 13 21:37:47.846400 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.846380 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:37:47.853277 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:47.853250 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c6c469-openshift-default-79c7cc8cff-qsnzw"] Jul 13 21:37:48.834876 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:48.834847 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" path="/var/lib/kubelet/pods/207a1629-ffc7-40c8-b687-c3f7916710e0/volumes" Jul 13 21:37:51.155878 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.155848 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:37:51.156260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156246 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.156260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156256 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.156345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156267 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:37:51.156345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156274 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:37:51.156345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156287 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.156345 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156294 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.156484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156370 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:37:51.156484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156380 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:37:51.156484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156387 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.156484 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.156393 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="207a1629-ffc7-40c8-b687-c3f7916710e0" containerName="istio-proxy" Jul 13 21:37:51.160922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.160905 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.164380 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.164362 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-18e186-openshift-default-dockercfg-567nz\"" Jul 13 21:37:51.178505 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.178483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:37:51.321738 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321694 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.321964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321781 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.321964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321814 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.321964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321841 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.321964 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.322140 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.321985 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.322140 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.322003 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.322140 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.322040 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.322140 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.322072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbk2\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.422762 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.422762 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422727 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422818 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422878 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422907 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423056 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.422990 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbk2\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423493 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.423139 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423493 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.423144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423493 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.423224 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423493 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.423279 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.423708 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.423650 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.425405 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.425377 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.425532 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.425430 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.431288 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.431267 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbk2\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.431394 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.431301 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token\") pod \"e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.471286 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.471262 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:51.622109 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.622081 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:37:51.623705 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:37:51.623677 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8f9671_8e8f_4285_af34_7b7e8946e407.slice/crio-c64d01558955f7f826f7c600ac96758d571cf74b8b5195b018d129cd85b50250 WatchSource:0}: Error finding container c64d01558955f7f826f7c600ac96758d571cf74b8b5195b018d129cd85b50250: Status 404 returned error can't find the container with id c64d01558955f7f826f7c600ac96758d571cf74b8b5195b018d129cd85b50250 Jul 13 21:37:51.626211 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.626180 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:37:51.626300 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.626240 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:37:51.626300 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.626273 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:37:51.847193 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.847158 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerStarted","Data":"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11"} Jul 13 21:37:51.847353 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.847214 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerStarted","Data":"c64d01558955f7f826f7c600ac96758d571cf74b8b5195b018d129cd85b50250"} Jul 13 21:37:51.868230 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:51.868182 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podStartSLOduration=0.868167216 podStartE2EDuration="868.167216ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:37:51.865723832 +0000 UTC m=+2647.594783757" watchObservedRunningTime="2026-07-13 21:37:51.868167216 +0000 UTC m=+2647.597227115" Jul 13 21:37:52.472164 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:52.472130 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:52.476903 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:52.476875 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:52.851784 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:52.851714 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:37:52.852583 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:37:52.852564 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:38:04.926875 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.926773 2561 generic.go:356] "Generic (PLEG): container finished" podID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerID="7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11" exitCode=0 Jul 13 21:38:04.926875 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.926858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerDied","Data":"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11"} Jul 13 21:38:04.927321 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.927260 2561 scope.go:117] "RemoveContainer" containerID="7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11" Jul 13 21:38:04.928312 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.928282 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:38:04.928412 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.928357 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:38:04.928412 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:04.928391 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 13 21:38:05.934453 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:05.934414 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerStarted","Data":"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243"} Jul 13 21:38:06.471978 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:06.471938 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:38:06.473131 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:06.473104 2561 patch_prober.go:28] interesting pod/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 13 21:38:06.473244 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:06.473162 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 13 21:38:07.471960 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:07.471919 2561 patch_prober.go:28] interesting pod/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 13 21:38:07.472439 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:07.471990 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 13 21:38:07.852279 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:07.852252 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:38:08.472520 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:08.472475 2561 patch_prober.go:28] interesting pod/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 13 21:38:08.472967 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:08.472535 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 13 21:38:09.472080 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:09.472040 2561 patch_prober.go:28] interesting pod/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 13 21:38:09.472269 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:09.472095 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 13 21:38:10.475984 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:10.475905 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:38:10.476897 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:10.476877 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:38:44.944687 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:44.944601 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:38:44.956678 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:38:44.956656 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:43:44.977190 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:43:44.977160 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:43:44.993423 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:43:44.993397 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n8svj_bd815d21-6e0e-464d-b647-e8e30d093c30/ovn-acl-logging/0.log" Jul 13 21:44:01.587881 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.587846 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs"] Jul 13 21:44:01.588290 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.588276 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:44:01.588290 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.588289 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbeef55-d0fb-4d79-8fc5-c0b98576b080" containerName="istio-proxy" Jul 13 21:44:01.591746 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.591728 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:01.594469 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.594444 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rnndk\"" Jul 13 21:44:01.597672 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.597649 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs"] Jul 13 21:44:01.674106 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.674081 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs\" (UID: \"00a43f5c-fe91-478e-8f2e-6be153c7c151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:01.776782 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.775889 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs\" (UID: \"00a43f5c-fe91-478e-8f2e-6be153c7c151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:01.780779 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.776964 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs\" (UID: \"00a43f5c-fe91-478e-8f2e-6be153c7c151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:01.904497 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:01.904464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:02.245730 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:02.245706 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs"] Jul 13 21:44:02.249238 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:44:02.248672 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a43f5c_fe91_478e_8f2e_6be153c7c151.slice/crio-6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51 WatchSource:0}: Error finding container 6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51: Status 404 returned error can't find the container with id 6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51 Jul 13 21:44:02.836375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:02.836289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" event={"ID":"00a43f5c-fe91-478e-8f2e-6be153c7c151","Type":"ContainerStarted","Data":"d3223699cb0ce36dbc595d698d054b9ef459fb253a479f300f57e66de92040b9"} Jul 13 21:44:02.836375 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:02.836343 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" event={"ID":"00a43f5c-fe91-478e-8f2e-6be153c7c151","Type":"ContainerStarted","Data":"6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51"} Jul 13 21:44:02.851765 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:02.851709 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" podStartSLOduration=1.8516968089999999 podStartE2EDuration="1.851696809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:44:02.85008164 +0000 UTC m=+3018.579141539" watchObservedRunningTime="2026-07-13 21:44:02.851696809 +0000 UTC m=+3018.580756708" Jul 13 21:44:03.842563 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:03.842531 2561 generic.go:356] "Generic (PLEG): container finished" podID="00a43f5c-fe91-478e-8f2e-6be153c7c151" containerID="d3223699cb0ce36dbc595d698d054b9ef459fb253a479f300f57e66de92040b9" exitCode=0 Jul 13 21:44:03.842969 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:03.842613 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" event={"ID":"00a43f5c-fe91-478e-8f2e-6be153c7c151","Type":"ContainerDied","Data":"d3223699cb0ce36dbc595d698d054b9ef459fb253a479f300f57e66de92040b9"} Jul 13 21:44:04.983711 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:04.983687 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:05.005428 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.005390 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca\") pod \"00a43f5c-fe91-478e-8f2e-6be153c7c151\" (UID: \"00a43f5c-fe91-478e-8f2e-6be153c7c151\") " Jul 13 21:44:05.005710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.005687 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00a43f5c-fe91-478e-8f2e-6be153c7c151" (UID: "00a43f5c-fe91-478e-8f2e-6be153c7c151"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:44:05.106675 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.106601 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a43f5c-fe91-478e-8f2e-6be153c7c151-maas-api-service-ca\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:05.853638 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.853602 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" event={"ID":"00a43f5c-fe91-478e-8f2e-6be153c7c151","Type":"ContainerDied","Data":"6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51"} Jul 13 21:44:05.853638 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.853620 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs" Jul 13 21:44:05.853888 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:05.853649 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce6146f0d13103da4d647ad7d9bb74e4fcbca17db316439139c5f27be431d51" Jul 13 21:44:53.456904 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:53.456811 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:44:53.457444 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:53.457105 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" containerID="cri-o://68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243" gracePeriod=30 Jul 13 21:44:55.478664 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:55.478630 2561 patch_prober.go:28] interesting pod/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 21:44:55.479147 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:55.478691 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 21:44:58.712115 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.712089 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:44:58.799576 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799576 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799527 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799576 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799548 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799588 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799628 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799646 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799670 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799710 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhbk2\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799738 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token\") pod \"5a8f9671-8e8f-4285-af34-7b7e8946e407\" (UID: \"5a8f9671-8e8f-4285-af34-7b7e8946e407\") " Jul 13 21:44:58.799928 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799828 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:44:58.800256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.799996 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:44:58.800256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.800045 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-credential-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.800256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.800042 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:44:58.800256 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.800030 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 21:44:58.800945 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.800914 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:44:58.801886 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.801865 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 21:44:58.801988 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.801875 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 21:44:58.802059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.802012 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:44:58.802059 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.802041 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2" (OuterVolumeSpecName: "kube-api-access-xhbk2") pod "5a8f9671-8e8f-4285-af34-7b7e8946e407" (UID: "5a8f9671-8e8f-4285-af34-7b7e8946e407"). InnerVolumeSpecName "kube-api-access-xhbk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 21:44:58.901144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901110 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-envoy\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901136 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-data\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901147 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-certs\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901156 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a8f9671-8e8f-4285-af34-7b7e8946e407-workload-socket\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901185 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbk2\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-kube-api-access-xhbk2\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901195 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-token\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901203 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a8f9671-8e8f-4285-af34-7b7e8946e407-istiod-ca-cert\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:58.901341 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:58.901210 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a8f9671-8e8f-4285-af34-7b7e8946e407-istio-podinfo\") on node \"ip-10-0-130-205.ec2.internal\" DevicePath \"\"" Jul 13 21:44:59.132296 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.132259 2561 generic.go:356] "Generic (PLEG): container finished" podID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerID="68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243" exitCode=0 Jul 13 21:44:59.132494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.132319 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" Jul 13 21:44:59.132494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.132349 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerDied","Data":"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243"} Jul 13 21:44:59.132494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.132416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44" event={"ID":"5a8f9671-8e8f-4285-af34-7b7e8946e407","Type":"ContainerDied","Data":"c64d01558955f7f826f7c600ac96758d571cf74b8b5195b018d129cd85b50250"} Jul 13 21:44:59.132494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.132448 2561 scope.go:117] "RemoveContainer" containerID="68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243" Jul 13 21:44:59.141628 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.141616 2561 scope.go:117] "RemoveContainer" containerID="7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11" Jul 13 21:44:59.149833 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.149813 2561 scope.go:117] "RemoveContainer" containerID="68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243" Jul 13 21:44:59.150111 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:44:59.150081 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243\": container with ID starting with 68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243 not found: ID does not exist" containerID="68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243" Jul 13 21:44:59.150213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.150118 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243"} err="failed to get container status \"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243\": rpc error: code = NotFound desc = could not find container \"68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243\": container with ID starting with 68cc4c485ec2cc69e4a0c7808d4c7fb8a164b13a15c333a99608a1e3541b3243 not found: ID does not exist" Jul 13 21:44:59.150213 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.150141 2561 scope.go:117] "RemoveContainer" containerID="7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11" Jul 13 21:44:59.150401 ip-10-0-130-205 kubenswrapper[2561]: E0713 21:44:59.150379 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11\": container with ID starting with 7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11 not found: ID does not exist" containerID="7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11" Jul 13 21:44:59.150462 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.150412 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11"} err="failed to get container status \"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11\": rpc error: code = NotFound desc = could not find container \"7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11\": container with ID starting with 7ef7ac5b7337c8f3c4e922f2571888ab05ccaea514e6d028acdc544adb83df11 not found: ID does not exist" Jul 13 21:44:59.150578 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.150562 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:44:59.153713 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.153693 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-18e186-openshift-default-f9bc48d8c-4px44"] Jul 13 21:44:59.215919 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.215894 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wd5g6_42559707-66f4-45dd-87b7-3e65885029e5/manager/0.log" Jul 13 21:44:59.335891 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.335862 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fbn97_b46fdf6d-e08e-4632-a577-015ef20c7b43/manager/0.log" Jul 13 21:44:59.459914 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.459837 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6dc4cf54-x9twb_edf8fe23-9275-461c-b729-16bff5ef3714/manager/0.log" Jul 13 21:44:59.584742 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.584716 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5w8tn_6cb76ffd-f6ec-4931-9f90-6e386b61c177/server/0.log" Jul 13 21:44:59.703445 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:44:59.703416 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6dzzx_24da7a97-e8e1-4bdb-a632-705bfbc59d8b/manager/0.log" Jul 13 21:45:00.064432 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:00.064401 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdc557c48-zsjvv_b4c38a8b-204e-4819-aac1-05d02fd8abe2/copy-manifests/0.log" Jul 13 21:45:00.077476 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:00.077448 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdc557c48-zsjvv_b4c38a8b-204e-4819-aac1-05d02fd8abe2/manager/0.log" Jul 13 21:45:00.434746 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:00.434716 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6ff844954-j5st8_d71a8cd7-2a6d-4c3e-a761-55ac5ac76d0d/maas-api/0.log" Jul 13 21:45:00.555210 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:00.555188 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732955-9hq6n_655ab8b2-e8be-4618-93fb-0cecd692913b/cleanup/0.log" Jul 13 21:45:00.834803 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:00.834708 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" path="/var/lib/kubelet/pods/5a8f9671-8e8f-4285-af34-7b7e8946e407/volumes" Jul 13 21:45:01.000710 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:01.000685 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-74bf51e2-xkx7c_f24ddb3c-c1d1-4383-beab-ab08e7042c6f/revoke-keys/0.log" Jul 13 21:45:01.441260 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:01.441232 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-613d27-tjlf2_8d12fa30-f5ba-4f42-bb1c-c04f7affed31/revoke-keys/0.log" Jul 13 21:45:02.004510 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:02.004478 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-0e15bb-9rfzs_00a43f5c-fe91-478e-8f2e-6be153c7c151/revoke-keys/0.log" Jul 13 21:45:02.225202 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:02.225171 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-t8pjg_801ece9b-849a-4dfd-acaf-5a4c39cd2b01/postgres/0.log" Jul 13 21:45:03.592011 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:03.591981 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qk4xj_44caa46c-8b5f-4304-be6a-83cd9b06cf23/manager/0.log" Jul 13 21:45:03.703225 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:03.703201 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cs4g5_74772e18-ece0-48d3-adb1-af8b082f1dbc/manager/0.log" Jul 13 21:45:03.923669 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:03.923645 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6lmxh_2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba/registry-server/0.log" Jul 13 21:45:04.053704 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:04.053678 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2gw46_a268a4a4-a21a-4abf-9ab5-f18760002309/manager/0.log" Jul 13 21:45:04.297169 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:04.297095 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zjvfb_eb0d2789-0097-416c-b489-9d0ae55403c8/manager/0.log" Jul 13 21:45:04.868125 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:04.868096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bf978c8f5-92btj_9028054c-8b61-48a9-92ee-c9d6aaf2add1/kube-auth-proxy/0.log" Jul 13 21:45:05.090527 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:05.090496 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/1.log" Jul 13 21:45:05.214325 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:05.214239 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-pswft_85dc4ad8-de67-47d1-bd81-06a1998c80a6/payload-pre-processing/0.log" Jul 13 21:45:05.329980 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:05.329948 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-zjxzg_c7ba5f09-81dc-4ea0-b0e9-4439fc98594a/payload-processing/0.log" Jul 13 21:45:05.913729 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:05.913699 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-krl2d_d62bab2a-cc2e-48fb-a46c-681f9cefa88f/main/0.log" Jul 13 21:45:06.138232 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:06.138203 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdjd5m_8546adf3-0b06-42c2-8375-c6eb6555bdc3/main/0.log" Jul 13 21:45:13.112021 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:13.111992 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bf8nw_03300b5c-abf8-4699-afa0-86ad3a8764c6/global-pull-secret-syncer/0.log" Jul 13 21:45:13.261847 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:13.261819 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5v6b_c610b0f5-60ef-494c-8810-0b9ab64a0df1/konnectivity-agent/0.log" Jul 13 21:45:13.287922 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:13.287890 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-205.ec2.internal_398ce9638757b84ca9a0be66ab3c6445/haproxy/0.log" Jul 13 21:45:17.917670 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:17.917574 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qk4xj_44caa46c-8b5f-4304-be6a-83cd9b06cf23/manager/0.log" Jul 13 21:45:17.966489 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:17.966459 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cs4g5_74772e18-ece0-48d3-adb1-af8b082f1dbc/manager/0.log" Jul 13 21:45:18.046074 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:18.046045 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6lmxh_2a6bf8b7-d493-4ba3-a802-66dc68ebe2ba/registry-server/0.log" Jul 13 21:45:18.179580 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:18.179500 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2gw46_a268a4a4-a21a-4abf-9ab5-f18760002309/manager/0.log" Jul 13 21:45:18.297596 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:18.297563 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zjvfb_eb0d2789-0097-416c-b489-9d0ae55403c8/manager/0.log" Jul 13 21:45:20.105706 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.105679 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-5c9lj_38d4da58-88f3-4733-992a-556b1e6ebc34/monitoring-plugin/0.log" Jul 13 21:45:20.139849 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.139816 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62chb_7e464e29-fb88-4184-a656-6ca19eb4d4ab/node-exporter/0.log" Jul 13 21:45:20.165265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.165245 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62chb_7e464e29-fb88-4184-a656-6ca19eb4d4ab/kube-rbac-proxy/0.log" Jul 13 21:45:20.189373 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.189348 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62chb_7e464e29-fb88-4184-a656-6ca19eb4d4ab/init-textfile/0.log" Jul 13 21:45:20.554433 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.554404 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/prometheus/0.log" Jul 13 21:45:20.578451 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.578371 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/config-reloader/0.log" Jul 13 21:45:20.601146 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.601127 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/thanos-sidecar/0.log" Jul 13 21:45:20.625180 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.625159 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/kube-rbac-proxy-web/0.log" Jul 13 21:45:20.652819 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.652800 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/kube-rbac-proxy/0.log" Jul 13 21:45:20.683549 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.683521 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/kube-rbac-proxy-thanos/0.log" Jul 13 21:45:20.708972 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.708928 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9543bcee-dc6a-4fb5-a9df-0dc3d93109e2/init-config-reloader/0.log" Jul 13 21:45:20.784114 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:20.784092 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-hmknq_8cc7bc47-368b-4ad9-a2f4-1f10f6e427cf/prometheus-operator-admission-webhook/0.log" Jul 13 21:45:22.251366 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:22.251339 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-hx2s6_dadf0e19-20a3-4890-bab4-7a2a9b1c147d/networking-console-plugin/0.log" Jul 13 21:45:23.258902 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.258871 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt"] Jul 13 21:45:23.259340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259325 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.259340 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259341 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.259430 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259352 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.259430 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259362 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.259430 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259374 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a43f5c-fe91-478e-8f2e-6be153c7c151" containerName="revoke-keys" Jul 13 21:45:23.259430 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259380 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a43f5c-fe91-478e-8f2e-6be153c7c151" containerName="revoke-keys" Jul 13 21:45:23.259575 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259470 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a43f5c-fe91-478e-8f2e-6be153c7c151" containerName="revoke-keys" Jul 13 21:45:23.259575 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259478 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.259575 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.259495 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8f9671-8e8f-4285-af34-7b7e8946e407" containerName="istio-proxy" Jul 13 21:45:23.264672 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.264653 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.266982 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.266962 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sxm8t\"/\"kube-root-ca.crt\"" Jul 13 21:45:23.267974 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.267954 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sxm8t\"/\"openshift-service-ca.crt\"" Jul 13 21:45:23.268102 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.267952 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sxm8t\"/\"default-dockercfg-4dwcg\"" Jul 13 21:45:23.270994 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.270972 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt"] Jul 13 21:45:23.416625 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.416596 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-lib-modules\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.416810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.416657 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-podres\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.416810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.416708 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-proc\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.416810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.416724 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-sys\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.416810 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.416747 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bdj\" (UniqueName: \"kubernetes.io/projected/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-kube-api-access-52bdj\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.453346 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.453322 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ccc957564-4s5rt_ed09fb08-a445-40a1-a1fa-e56feda559f7/console/0.log" Jul 13 21:45:23.492135 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.492108 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-nvct7_a27239ed-2691-42d5-b547-3ef63e441ce9/download-server/0.log" Jul 13 21:45:23.517494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517429 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-podres\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517468 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-proc\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517494 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517486 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-sys\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-sys\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517551 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bdj\" (UniqueName: \"kubernetes.io/projected/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-kube-api-access-52bdj\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517613 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-podres\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517557 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-proc\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517645 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-lib-modules\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.517787 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.517776 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-lib-modules\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.524830 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.524812 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bdj\" (UniqueName: \"kubernetes.io/projected/14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4-kube-api-access-52bdj\") pod \"perf-node-gather-daemonset-rvlxt\" (UID: \"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4\") " pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.575944 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.575914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:23.709092 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:23.709064 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt"] Jul 13 21:45:23.710980 ip-10-0-130-205 kubenswrapper[2561]: W0713 21:45:23.710949 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14f28ddb_43ad_4ccf_9bf6_c2fb6f8cfba4.slice/crio-f6da1df0fe9bb1a5b0b3a0d6259d6c152325429a66bb52132a8250c15d51a5a5 WatchSource:0}: Error finding container f6da1df0fe9bb1a5b0b3a0d6259d6c152325429a66bb52132a8250c15d51a5a5: Status 404 returned error can't find the container with id f6da1df0fe9bb1a5b0b3a0d6259d6c152325429a66bb52132a8250c15d51a5a5 Jul 13 21:45:24.271000 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:24.270964 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" event={"ID":"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4","Type":"ContainerStarted","Data":"cd4f003cf0a48c81d96b88507a1979532343f04338e4eda566f35c5a7a99599b"} Jul 13 21:45:24.271401 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:24.271022 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" event={"ID":"14f28ddb-43ad-4ccf-9bf6-c2fb6f8cfba4","Type":"ContainerStarted","Data":"f6da1df0fe9bb1a5b0b3a0d6259d6c152325429a66bb52132a8250c15d51a5a5"} Jul 13 21:45:24.271401 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:24.271105 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:24.288117 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:24.288067 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" podStartSLOduration=1.2880531 podStartE2EDuration="1.2880531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:45:24.285641303 +0000 UTC m=+3100.014701219" watchObservedRunningTime="2026-07-13 21:45:24.2880531 +0000 UTC m=+3100.017112999" Jul 13 21:45:25.196181 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:25.196153 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-252h7_9ce9d425-a05e-4431-9c38-d928e5728715/dns/0.log" Jul 13 21:45:25.229880 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:25.229854 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-252h7_9ce9d425-a05e-4431-9c38-d928e5728715/kube-rbac-proxy/0.log" Jul 13 21:45:25.402994 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:25.402967 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v5cd5_bbeddf55-2119-4a5b-94be-b776003f0a3c/dns-node-resolver/0.log" Jul 13 21:45:25.996257 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:25.996228 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8b9q9_e9798223-c3c2-4b30-8311-e9e6780dde4d/node-ca/0.log" Jul 13 21:45:27.094992 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:27.094955 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bf978c8f5-92btj_9028054c-8b61-48a9-92ee-c9d6aaf2add1/kube-auth-proxy/0.log" Jul 13 21:45:27.146051 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:27.146024 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/0.log" Jul 13 21:45:27.174069 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:27.174044 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/1.log" Jul 13 21:45:27.204144 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:27.204122 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-pswft_85dc4ad8-de67-47d1-bd81-06a1998c80a6/payload-pre-processing/0.log" Jul 13 21:45:27.233936 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:27.233914 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-zjxzg_c7ba5f09-81dc-4ea0-b0e9-4439fc98594a/payload-processing/0.log" Jul 13 21:45:28.015465 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:28.015441 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bgs6t_1833b755-6f8b-4268-b9c9-0471a1755708/serve-healthcheck-canary/0.log" Jul 13 21:45:28.528958 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:28.528924 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27wmt_17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a/kube-rbac-proxy/0.log" Jul 13 21:45:28.550791 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:28.550742 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27wmt_17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a/exporter/0.log" Jul 13 21:45:28.573094 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:28.573073 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-27wmt_17f9265c-0aec-4eb5-9e2a-df92bcf1ef1a/extractor/0.log" Jul 13 21:45:30.286122 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:30.286092 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sxm8t/perf-node-gather-daemonset-rvlxt" Jul 13 21:45:30.887512 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:30.887484 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-wd5g6_42559707-66f4-45dd-87b7-3e65885029e5/manager/0.log" Jul 13 21:45:31.024764 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.024734 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fbn97_b46fdf6d-e08e-4632-a577-015ef20c7b43/manager/0.log" Jul 13 21:45:31.125128 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.125096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f6dc4cf54-x9twb_edf8fe23-9275-461c-b729-16bff5ef3714/manager/0.log" Jul 13 21:45:31.154769 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.154684 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5w8tn_6cb76ffd-f6ec-4931-9f90-6e386b61c177/server/0.log" Jul 13 21:45:31.220790 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.220766 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6dzzx_24da7a97-e8e1-4bdb-a632-705bfbc59d8b/manager/0.log" Jul 13 21:45:31.394187 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.394152 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdc557c48-zsjvv_b4c38a8b-204e-4819-aac1-05d02fd8abe2/manager/0.log" Jul 13 21:45:31.412925 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:31.412855 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fdc557c48-zsjvv_b4c38a8b-204e-4819-aac1-05d02fd8abe2/copy-manifests/0.log" Jul 13 21:45:33.771089 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:33.771011 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bf978c8f5-92btj_9028054c-8b61-48a9-92ee-c9d6aaf2add1/kube-auth-proxy/0.log" Jul 13 21:45:33.805871 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:33.805815 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/0.log" Jul 13 21:45:33.835770 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:33.835723 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/1.log" Jul 13 21:45:33.861686 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:33.861658 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-pswft_85dc4ad8-de67-47d1-bd81-06a1998c80a6/payload-pre-processing/0.log" Jul 13 21:45:33.882503 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:33.882479 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-zjxzg_c7ba5f09-81dc-4ea0-b0e9-4439fc98594a/payload-processing/0.log" Jul 13 21:45:35.302967 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:35.302938 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bf978c8f5-92btj_9028054c-8b61-48a9-92ee-c9d6aaf2add1/kube-auth-proxy/0.log" Jul 13 21:45:35.339546 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:35.339469 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/0.log" Jul 13 21:45:35.369235 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:35.369185 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-q8vsw_71ac7e7c-12a7-4050-84a0-023a08a6f1a2/istio-proxy/1.log" Jul 13 21:45:35.390265 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:35.390240 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-pswft_85dc4ad8-de67-47d1-bd81-06a1998c80a6/payload-pre-processing/0.log" Jul 13 21:45:35.411402 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:35.411375 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-zjxzg_c7ba5f09-81dc-4ea0-b0e9-4439fc98594a/payload-processing/0.log" Jul 13 21:45:37.997255 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:37.997200 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dn9r9_a6257856-68e0-4dd0-bda7-161972d9bc9f/migrator/0.log" Jul 13 21:45:38.021073 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:38.021043 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-dn9r9_a6257856-68e0-4dd0-bda7-161972d9bc9f/graceful-termination/0.log" Jul 13 21:45:39.586335 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.586267 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/kube-multus-additional-cni-plugins/0.log" Jul 13 21:45:39.607424 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.607401 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/egress-router-binary-copy/0.log" Jul 13 21:45:39.628063 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.628045 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/cni-plugins/0.log" Jul 13 21:45:39.652034 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.652013 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/bond-cni-plugin/0.log" Jul 13 21:45:39.673485 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.673468 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/routeoverride-cni/0.log" Jul 13 21:45:39.694313 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.694292 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/whereabouts-cni-bincopy/0.log" Jul 13 21:45:39.715464 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.715447 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mxnmh_f89949d5-9982-45e3-be0c-b984d9babe6c/whereabouts-cni/0.log" Jul 13 21:45:39.959881 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.959855 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vsrzm_58293a86-df78-4dae-9e33-71394395d584/kube-multus/0.log" Jul 13 21:45:39.981727 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:39.981701 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-97vfm_2c838c29-7ff1-4577-ad7e-209822026fb9/network-metrics-daemon/0.log" Jul 13 21:45:40.001546 ip-10-0-130-205 kubenswrapper[2561]: I0713 21:45:40.001518 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-97vfm_2c838c29-7ff1-4577-ad7e-209822026fb9/kube-rbac-proxy/0.log"