Jul 15 23:30:10.588472 ip-10-0-142-149 systemd[1]: Starting Kubernetes Kubelet... Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.966762 2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975079 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975112 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975129 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975244 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975257 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975265 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975269 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.092065 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975274 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.051546 ip-10-0-142-149 systemd[1]: Started Kubernetes Kubelet. Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975279 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975283 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975287 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975293 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975298 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975302 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975307 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975311 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975316 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975321 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975327 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975334 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975338 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975343 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975347 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975352 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975356 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975372 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975376 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.093394 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975381 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975386 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975390 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975394 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975399 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975403 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975411 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975417 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975421 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975425 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975432 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975439 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975444 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975449 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975454 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975459 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975463 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975467 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975470 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.094282 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975472 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975475 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975478 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975481 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975485 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975488 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975491 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975494 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975497 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975499 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975502 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975505 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975508 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975511 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975514 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975517 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975519 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975522 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975526 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975528 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.094953 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975532 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975535 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975538 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975541 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975543 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975547 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975550 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975553 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975556 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975559 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975562 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975566 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975568 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975571 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975576 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975579 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975582 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975585 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975588 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.095687 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975591 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975594 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975597 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975599 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975602 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975606 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975609 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975612 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975615 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975618 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975621 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975624 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975627 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.975629 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976511 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976521 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976524 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976527 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976531 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976534 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.096338 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976537 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976540 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976543 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976546 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976549 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976552 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976557 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976561 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976565 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976568 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976571 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976575 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976578 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976581 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976584 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976587 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976590 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976593 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976597 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.097001 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976600 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976603 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976605 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976609 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976612 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976615 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976624 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976627 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976630 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976633 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976636 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976638 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976641 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976644 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976647 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976650 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976653 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976656 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976658 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976662 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.097808 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976664 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976667 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976670 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976673 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976676 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976679 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976682 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976685 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976688 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976691 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976693 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976696 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976699 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976701 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976705 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976714 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976717 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976720 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976725 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976728 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.098730 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976730 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976733 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976736 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976739 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976742 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976745 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976747 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976750 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976752 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976757 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976761 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976764 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976768 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976771 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976773 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976778 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976782 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976785 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976789 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.099450 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976793 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976796 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976799 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976802 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976805 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976809 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976812 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976815 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976818 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976821 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976824 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976828 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976831 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976834 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.976837 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977340 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977347 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977351 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977355 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.100081 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977358 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977361 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977365 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977368 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977371 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977374 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977377 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977380 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977383 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977385 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977388 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977390 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977394 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977399 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977403 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977406 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977409 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977412 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977415 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.100785 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977418 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977421 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977424 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977427 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977430 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977434 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977438 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977441 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977444 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977447 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977450 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977452 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977456 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977458 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977461 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977464 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977466 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977470 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977473 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977475 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.101425 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977478 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977481 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977484 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977487 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977489 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977492 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977497 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977500 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977504 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977507 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977510 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977513 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977517 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977520 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977523 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977525 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977528 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977533 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977536 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977539 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.102145 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977542 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977545 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977548 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977551 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977553 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977556 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977558 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977561 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977564 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977566 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977569 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977572 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977575 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977578 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977581 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977584 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977587 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977589 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977594 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977598 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.102830 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977602 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977606 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977609 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977611 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977614 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977617 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977619 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977622 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977625 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977630 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977633 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977635 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977638 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977641 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977644 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.977647 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977733 2551 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977752 2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977763 2551 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977770 2551 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 23:30:11.103525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977775 2551 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977779 2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977786 2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977791 2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977795 2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977799 2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977803 2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977806 2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977810 2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977813 2551 flags.go:64] FLAG: --cgroup-root="" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977816 2551 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977819 2551 flags.go:64] FLAG: --client-ca-file="" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977824 2551 flags.go:64] FLAG: --cloud-provider="external" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977828 2551 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977833 2551 flags.go:64] FLAG: --cluster-domain="" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977836 2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977839 2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977843 2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977846 2551 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977850 2551 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977854 2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977859 2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977862 2551 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977866 2551 flags.go:64] FLAG: --contention-profiling="false" Jul 15 23:30:11.104118 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977869 2551 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977873 2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977876 2551 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977880 2551 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977885 2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977888 2551 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977892 2551 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977895 2551 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977898 2551 flags.go:64] FLAG: --enable-server="true" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977902 2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977909 2551 flags.go:64] FLAG: --event-burst="100" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977913 2551 flags.go:64] FLAG: --event-qps="50" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977916 2551 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977919 2551 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977922 2551 flags.go:64] FLAG: --eviction-hard="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977927 2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977930 2551 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977933 2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977937 2551 flags.go:64] FLAG: --eviction-soft="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977940 2551 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977943 2551 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977947 2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977951 2551 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977954 2551 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977957 2551 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 23:30:11.104860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977961 2551 flags.go:64] FLAG: --feature-gates="" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977966 2551 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977969 2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977972 2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977977 2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977981 2551 flags.go:64] FLAG: --healthz-port="10248" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977984 2551 flags.go:64] FLAG: --help="false" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977987 2551 flags.go:64] FLAG: --hostname-override="ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977991 2551 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977994 2551 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.977997 2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978001 2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978005 2551 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978008 2551 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978011 2551 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978015 2551 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978018 2551 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978023 2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978026 2551 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978029 2551 flags.go:64] FLAG: --kube-reserved="" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978033 2551 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978036 2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978039 2551 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978043 2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 23:30:11.105688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978046 2551 flags.go:64] FLAG: --lock-file="" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978049 2551 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978052 2551 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978055 2551 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978061 2551 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978066 2551 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978069 2551 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978072 2551 flags.go:64] FLAG: --logging-format="text" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978075 2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978079 2551 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978082 2551 flags.go:64] FLAG: --manifest-url="" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978085 2551 flags.go:64] FLAG: --manifest-url-header="" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978092 2551 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978096 2551 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978101 2551 flags.go:64] FLAG: --max-pods="110" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978104 2551 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978107 2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978110 2551 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978113 2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978117 2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978120 2551 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978123 2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978132 2551 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978135 2551 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978139 2551 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 23:30:11.108872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978143 2551 flags.go:64] FLAG: --pod-cidr="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978147 2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978152 2551 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978155 2551 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978159 2551 flags.go:64] FLAG: --pods-per-core="0" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978162 2551 flags.go:64] FLAG: --port="10250" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978165 2551 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978168 2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0790367ec48d53bfa" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978172 2551 flags.go:64] FLAG: --qos-reserved="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978175 2551 flags.go:64] FLAG: --read-only-port="10255" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978179 2551 flags.go:64] FLAG: --register-node="true" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978182 2551 flags.go:64] FLAG: --register-with-taints="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978186 2551 flags.go:64] FLAG: --registry-burst="10" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978190 2551 flags.go:64] FLAG: --registry-qps="5" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978193 2551 flags.go:64] FLAG: --reserved-cpus="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978212 2551 flags.go:64] FLAG: --reserved-memory="" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978217 2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978220 2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978224 2551 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978227 2551 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978232 2551 flags.go:64] FLAG: --runonce="false" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978235 2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978238 2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978242 2551 flags.go:64] FLAG: --seccomp-default="false" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978245 2551 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 23:30:11.110631 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978248 2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978252 2551 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978255 2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978259 2551 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978262 2551 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978265 2551 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978269 2551 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978273 2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978277 2551 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978280 2551 flags.go:64] FLAG: --system-cgroups="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978284 2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978290 2551 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978293 2551 flags.go:64] FLAG: --tls-cert-file="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978296 2551 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978301 2551 flags.go:64] FLAG: --tls-min-version="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978304 2551 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978307 2551 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978311 2551 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978314 2551 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978317 2551 flags.go:64] FLAG: --v="2" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978322 2551 flags.go:64] FLAG: --version="false" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978327 2551 flags.go:64] FLAG: --vmodule="" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978332 2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.978335 2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978470 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.111491 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978474 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978477 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978482 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978485 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978487 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978490 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978494 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978497 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978500 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978504 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978508 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978511 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978514 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978517 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978521 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978524 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978527 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978530 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978532 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978535 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.112426 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978538 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978541 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978544 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978547 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978549 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978552 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978555 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978558 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978560 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978564 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978567 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978569 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978572 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978575 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978580 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978584 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978588 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978591 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978595 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.113149 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978598 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978601 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978604 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978607 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978610 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978614 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978617 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978621 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978624 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978627 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978630 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978633 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978635 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978638 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978641 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978643 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978646 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978649 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978651 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978654 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.304771 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978657 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978660 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978662 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978665 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978668 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978671 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978674 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978678 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978681 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978684 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978687 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978690 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978692 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978696 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978698 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978701 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978704 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978707 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978710 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978714 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.324185 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978717 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978721 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978725 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978728 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978731 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978735 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978738 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978741 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978744 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978747 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978749 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978753 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978756 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978759 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978762 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978764 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978768 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978770 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.325050 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.978773 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.979670 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.989364 2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.989391 2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.990418 2551 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.990430 2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990504 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990512 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990516 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990519 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990523 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990527 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990531 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990534 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990537 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.325704 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990541 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990543 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990547 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990550 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990553 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990556 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990559 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990561 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990564 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990567 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990571 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990574 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990578 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990581 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990583 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990586 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990589 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990592 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990595 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990597 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.326292 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990604 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990607 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990610 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990613 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990615 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990618 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990621 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990623 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990626 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990630 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990633 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990635 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990638 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990641 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990644 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990647 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990650 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990652 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990659 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990663 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.326952 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990666 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990672 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990675 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990678 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990681 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990684 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990687 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990690 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990693 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990696 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990698 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990701 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990704 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990708 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990712 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990715 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990717 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990722 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990726 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.327657 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990730 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990735 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990740 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990745 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990749 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990752 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990755 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990758 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990761 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990763 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990766 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990772 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990774 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990777 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990782 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990785 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990788 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990791 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.328285 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990795 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990798 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990800 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990803 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990806 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990809 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990813 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990816 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990819 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990823 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990826 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990829 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990832 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.990838 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990985 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 23:30:11.328895 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990990 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990993 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.990996 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991000 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991003 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991006 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991010 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991013 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991017 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991020 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991026 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991030 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991032 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991035 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991039 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991042 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991045 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991048 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991051 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991054 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 23:30:11.329427 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991057 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991060 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991063 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991067 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991070 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991073 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991076 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991079 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991082 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991085 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991087 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991090 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991093 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991095 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991098 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991101 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991104 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991106 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991109 2551 feature_gate.go:328] unrecognized feature gate: Example Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991112 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 23:30:11.330147 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991114 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991117 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991121 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991124 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991128 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991130 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991134 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991137 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991140 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991143 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991147 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991151 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991154 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991158 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991161 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991164 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991166 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991169 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991172 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991175 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 23:30:11.330805 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991179 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991200 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991205 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991208 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991212 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991215 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991219 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991222 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991225 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991228 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991231 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991234 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991237 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991240 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991244 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991247 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991251 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991255 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991259 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 23:30:11.331526 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991262 2551 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991265 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991268 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991271 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991274 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991277 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991280 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991283 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991286 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991289 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991292 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991295 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991298 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991301 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991305 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991308 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991310 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991313 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 23:30:11.332140 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:10.991316 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.991321 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.991489 2551 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.994105 2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.995691 2551 server.go:1019] "Starting client certificate rotation" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.995804 2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.995843 2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:10.999175 2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.000653 2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.014272 2551 log.go:25] "Validated CRI v1 runtime API" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.017433 2551 log.go:25] "Validated CRI v1 image API" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.018417 2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 23:30:11.332765 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.021259 2551 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d9f4786c-3f7f-4d73-aaf4-16f04fa047e5:/dev/nvme0n1p4 ef17cec0-a675-4ae7-9bf9-807845b27225:/dev/nvme0n1p3] Jul 15 23:30:11.333183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.021277 2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 23:30:11.333183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.024090 2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.027697 2551 manager.go:216] Machine: {Timestamp:2026-07-15 23:30:11.025649317 +0000 UTC m=+0.314464741 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100199 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2301f4fd8787110f7d4060dc55fe3f SystemUUID:ec2301f4-fd87-8711-0f7d-4060dc55fe3f BootID:32a54c37-5f21-461c-9978-37b9b8133c9c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:36:78:ca:59:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:36:78:ca:59:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:e5:b1:d9:cb:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.027821 2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.027924 2551 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.030457 2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.030491 2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.031271 2551 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.031283 2551 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.031291 2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.032649 2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.034263 2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037716 2551 state_mem.go:36] "Initialized new in-memory state store" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037911 2551 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037947 2551 kubelet.go:486] "Attempting to sync node with API server" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037959 2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037976 2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.037989 2551 kubelet.go:398] "Adding apiserver pod source" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.038004 2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 23:30:11.333445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.038557 2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.038581 2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.042112 2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.044705 2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.044741 2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.044995 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045010 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045018 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045023 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045032 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045040 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045459 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045467 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045479 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045486 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045496 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045505 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045530 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.045540 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.049265 2551 server.go:1268] "Started kubelet" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.049516 2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.049628 2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.049949 2551 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.051958 2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-twl5j" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.051961 2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.053130 2551 server.go:310] "Adding debug handlers to kubelet server" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.054050 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.054191 2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.060523 2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.060760 2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.060796 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.060903 2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-twl5j" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.061223 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.062088 2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.062404 2551 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.062418 2551 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.062672 2551 reconstruct.go:97] "Volume reconstruction finished" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.062681 2551 reconciler.go:29] "Reconciler: start to sync state" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.062997 2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.063587 2551 factory.go:55] Registering systemd factory Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.063608 2551 factory.go:223] Registration of the systemd container factory successfully Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064055 2551 factory.go:153] Registering CRI-O factory Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064074 2551 factory.go:223] Registration of the crio container factory successfully Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064136 2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064170 2551 factory.go:103] Registering Raw factory Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064300 2551 manager.go:1196] Started watching for new ooms in manager Jul 15 23:30:11.334183 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.064980 2551 manager.go:349] Starting recovery of all containers Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.068172 2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.068287 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.068429 2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-149.ec2.internal.18c29a70a8af9c37 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-149.ec2.internal,UID:ip-10-0-142-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-149.ec2.internal,},FirstTimestamp:2026-07-15 23:30:11.049233463 +0000 UTC m=+0.338048886,LastTimestamp:2026-07-15 23:30:11.049233463 +0000 UTC m=+0.338048886,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-149.ec2.internal,}" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.079262 2551 manager.go:354] Recovery completed Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.085469 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.087891 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.087929 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.087953 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.088643 2551 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.088657 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.088659 2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.088743 2551 state_mem.go:36] "Initialized new in-memory state store" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.090033 2551 policy_none.go:49] "None policy: Start" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.090694 2551 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.090718 2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.091979 2551 policy_none.go:47] "Start" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.092265 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.092303 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.092316 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140012 2551 manager.go:337] "Starting Device Plugin manager" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.140217 2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140232 2551 server.go:87] "Starting device plugin registration server" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140519 2551 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140531 2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140650 2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140743 2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.140753 2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.142059 2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.142124 2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.244682 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.245840 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.245879 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.245891 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.245921 2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.254393 2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.254424 2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-149.ec2.internal\": node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.275360 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.335624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.294811 2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 23:30:11.337572 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.296215 2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 23:30:11.337572 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.296262 2551 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 23:30:11.337572 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.296323 2551 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 23:30:11.337572 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.296364 2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 23:30:11.337572 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.299103 2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.375854 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.397625 2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal"] Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.397713 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.399760 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.399792 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.399806 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.401164 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.401725 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.401757 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402638 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402904 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402937 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402965 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402990 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.402967 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.405351 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.405394 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.406502 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.406528 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.406538 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.442541 2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-149.ec2.internal\" not found" node="ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.447974 2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-149.ec2.internal\" not found" node="ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.465793 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.465827 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.465849 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21088ddecb53e5296158fe55990a64c7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-149.ec2.internal\" (UID: \"21088ddecb53e5296158fe55990a64c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.505715 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.476308 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.565993 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.566051 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.566071 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21088ddecb53e5296158fe55990a64c7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-149.ec2.internal\" (UID: \"21088ddecb53e5296158fe55990a64c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.566118 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21088ddecb53e5296158fe55990a64c7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-149.ec2.internal\" (UID: \"21088ddecb53e5296158fe55990a64c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.566192 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.566221 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/290d0c9e2524a1d4503c0280421e2430-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal\" (UID: \"290d0c9e2524a1d4503c0280421e2430\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.576442 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.678830 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.677528 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:11.871913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.744756 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.871913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.752850 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:11.871913 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.778692 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.879331 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.975727 2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:11.980051 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.994910 2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.995065 2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:11.995098 2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.066078 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 23:25:11 +0000 UTC" deadline="2027-12-17 22:50:23.03924833 +0000 UTC" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.066110 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12479h20m10.973142107s" Jul 15 23:30:12.111083 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:12.080988 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.286345 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:12.182185 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.286345 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:12.283190 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.416948 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:12.383897 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-149.ec2.internal\" not found" Jul 15 23:30:12.416948 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.384732 2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:12.424998 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21088ddecb53e5296158fe55990a64c7.slice/crio-306074174d1112433b2a2190b336c0fa76afdb79412edc827ce3e1f8029d2796 WatchSource:0}: Error finding container 306074174d1112433b2a2190b336c0fa76afdb79412edc827ce3e1f8029d2796: Status 404 returned error can't find the container with id 306074174d1112433b2a2190b336c0fa76afdb79412edc827ce3e1f8029d2796 Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:12.425241 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290d0c9e2524a1d4503c0280421e2430.slice/crio-b88568ccda8bae6a18e12db34fead541093ed144291d52e23ac545b40b106e26 WatchSource:0}: Error finding container b88568ccda8bae6a18e12db34fead541093ed144291d52e23ac545b40b106e26: Status 404 returned error can't find the container with id b88568ccda8bae6a18e12db34fead541093ed144291d52e23ac545b40b106e26 Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.429281 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.433501 2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.462445 2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.479090 2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.480163 2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" Jul 15 23:30:12.574447 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:12.496686 2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 23:30:13.000375 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.000336 2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:13.041185 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.041151 2551 apiserver.go:52] "Watching apiserver" Jul 15 23:30:13.052469 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.052433 2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 23:30:13.053825 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.053791 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x5ftd","kube-system/konnectivity-agent-rlzmg","kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr","openshift-dns/node-resolver-zqc8n","openshift-multus/multus-nbfwm","openshift-multus/network-metrics-daemon-mggsj","openshift-network-operator/iptables-alerter-xzv2p","openshift-cluster-node-tuning-operator/tuned-h8xk2","openshift-image-registry/node-ca-gj5mm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal","openshift-multus/multus-additional-cni-plugins-djqxk","openshift-network-diagnostics/network-check-target-km96l","openshift-ovn-kubernetes/ovnkube-node-x4jhr"] Jul 15 23:30:13.056624 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.056596 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.059139 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.059112 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:13.059291 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.059246 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:13.060853 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.060826 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.060954 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.060895 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.061120 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.061098 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 23:30:13.061199 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.061134 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tm886\"" Jul 15 23:30:13.063409 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.063383 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.064889 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.064866 2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 23:30:13.065827 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.065806 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.066863 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.066840 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cw9lb\"" Jul 15 23:30:13.067000 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.066886 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 23:30:13.067114 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.067097 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 23:30:13.067171 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.067162 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.068137 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.067322 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.073380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073255 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 23:30:13.073380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073283 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 23:30:13.073380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073299 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rptgb\"" Jul 15 23:30:13.073653 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073475 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 23:30:13.073750 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073716 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.073804 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.073790 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 23:30:13.073854 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.073817 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:13.075102 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.075077 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.076386 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076359 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-ovn\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.076500 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076404 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-node-log\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.076500 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076440 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-log-socket\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.076500 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076468 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.076673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076503 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkm8\" (UniqueName: \"kubernetes.io/projected/ce3ad4a2-a07d-4f4e-b17f-387346234df0-kube-api-access-gtkm8\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.076673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076547 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plrgd\" (UniqueName: \"kubernetes.io/projected/b8f925a3-e179-41e3-af2e-1eb891af39be-kube-api-access-plrgd\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.076673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076581 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:13.076673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.076609 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-netns\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.077077 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.077053 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-etc-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.078053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.077859 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-netd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.078053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.077910 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-config\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.078053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.077971 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovn-node-metrics-cert\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080680 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tqbvf\"" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080734 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8f925a3-e179-41e3-af2e-1eb891af39be-host\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080756 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080774 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080780 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-systemd-units\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080866 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-systemd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080908 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080932 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-bin\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080958 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080986 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-env-overrides\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.080990 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081043 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081050 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-kubelet\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081087 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-slash\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081116 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-script-lib\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081146 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8f925a3-e179-41e3-af2e-1eb891af39be-serviceca\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.081362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081170 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-var-lib-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.082406 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.081509 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.083415 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.083387 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.084323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.084284 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.084323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.084308 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.084520 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.084332 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwkg6\"" Jul 15 23:30:13.084520 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.084396 2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 23:30:13.086015 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.085994 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.086536 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.086261 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.086673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.086617 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q4c4g\"" Jul 15 23:30:13.087097 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.087057 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.088712 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.088689 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.088819 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.088777 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:13.089932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.089912 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.090385 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.090366 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8wxgm\"" Jul 15 23:30:13.090597 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.090578 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 23:30:13.090750 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.090699 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.091653 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.091630 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.094759 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.093967 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.096282 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.096141 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 23:30:13.096282 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.096158 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kdj9b\"" Jul 15 23:30:13.096511 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.096493 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 23:30:13.097764 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.097643 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 23:30:13.099276 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.098508 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wd2ks\"" Jul 15 23:30:13.099276 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.098521 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:30:13.099276 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.098854 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 23:30:13.120038 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.120008 2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2wt8d" Jul 15 23:30:13.129370 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.129340 2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2wt8d" Jul 15 23:30:13.164456 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.164426 2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181495 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-socket-dir-parent\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181545 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181575 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-conf\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181602 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-lib-modules\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181630 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-bin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181674 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-kubelet\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181701 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2bp\" (UniqueName: \"kubernetes.io/projected/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-kube-api-access-vw2bp\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181728 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b9d180-ba25-43e8-9ddf-7a91608ea469-tmp-dir\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181762 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-ovn\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181788 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181819 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfsw\" (UniqueName: \"kubernetes.io/projected/f7b9d180-ba25-43e8-9ddf-7a91608ea469-kube-api-access-qhfsw\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181847 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181877 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-netd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.181998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.181932 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-systemd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182022 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-netd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182061 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-ovn\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182270 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182316 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-systemd\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182363 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-bin\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182396 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-system-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182461 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-agent-certs\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182482 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-cni-bin\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182495 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-host\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182548 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-kubelet\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182543 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-run-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182577 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-cnibin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182627 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-kubelet\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182662 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-cni-binary-copy\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182694 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-socket-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182730 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.182862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182774 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2j5k\" (UniqueName: \"kubernetes.io/projected/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-kube-api-access-z2j5k\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182819 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-sys\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182852 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-node-log\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182881 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182905 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182930 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-os-release\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182950 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-node-log\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182968 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-device-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.182993 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-konnectivity-ca\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183005 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183023 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysconfig\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183054 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-systemd\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183092 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-etc-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183134 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovn-node-metrics-cert\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183180 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-etc-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183226 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-conf-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183273 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-system-cni-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.183731 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183343 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8f925a3-e179-41e3-af2e-1eb891af39be-host\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183373 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-systemd-units\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183403 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-registration-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183446 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cnibin\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183472 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183497 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxbp\" (UniqueName: \"kubernetes.io/projected/c7329761-9c29-4771-a1f6-561c797a93ce-kube-api-access-8jxbp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183539 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-slash\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183565 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-etc-kubernetes\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183592 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183633 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-etc-selinux\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183658 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183663 2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183687 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-etc-tuned\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183717 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-tmp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183746 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8f925a3-e179-41e3-af2e-1eb891af39be-serviceca\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183775 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-var-lib-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183802 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-log-socket\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.184713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183827 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-kubernetes\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183850 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-run\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183883 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhd68\" (UniqueName: \"kubernetes.io/projected/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-kube-api-access-xhd68\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183915 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-config\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.183942 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-netns\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184041 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnllt\" (UniqueName: \"kubernetes.io/projected/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kube-api-access-bnllt\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184051 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-log-socket\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184070 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-kubelet-config\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184099 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-host-slash\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184117 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b8f925a3-e179-41e3-af2e-1eb891af39be-host\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184129 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-os-release\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184154 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-systemd-units\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184156 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-iptables-alerter-script\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184196 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-script-lib\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184298 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-k8s-cni-cncf-io\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184363 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-daemon-config\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184393 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-multus-certs\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.185640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184422 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-dbus\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184459 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7b9d180-ba25-43e8-9ddf-7a91608ea469-hosts-file\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184480 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-modprobe-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184516 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkm8\" (UniqueName: \"kubernetes.io/projected/ce3ad4a2-a07d-4f4e-b17f-387346234df0-kube-api-access-gtkm8\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184540 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-hostroot\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184576 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plrgd\" (UniqueName: \"kubernetes.io/projected/b8f925a3-e179-41e3-af2e-1eb891af39be-kube-api-access-plrgd\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184611 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-netns\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184642 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-multus\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184675 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq576\" (UniqueName: \"kubernetes.io/projected/d8b83ce7-2b96-4123-bae7-47993008f3b1-kube-api-access-lq576\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184704 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b8f925a3-e179-41e3-af2e-1eb891af39be-serviceca\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184773 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-sys-fs\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184782 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-var-lib-openvswitch\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184801 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184831 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184854 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-var-lib-kubelet\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184864 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-slash\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184881 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.186537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.184910 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-env-overrides\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.187502 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.185081 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-run-netns\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.187502 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.185479 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-script-lib\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.187502 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.185507 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovnkube-config\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.187502 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.185575 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce3ad4a2-a07d-4f4e-b17f-387346234df0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.187502 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.185878 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ce3ad4a2-a07d-4f4e-b17f-387346234df0-env-overrides\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.189684 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.189648 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce3ad4a2-a07d-4f4e-b17f-387346234df0-ovn-node-metrics-cert\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.190697 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.190671 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:13.190821 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.190706 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:13.190821 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.190723 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:13.190943 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.190854 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:13.690819169 +0000 UTC m=+2.979634597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:13.198519 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.198486 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plrgd\" (UniqueName: \"kubernetes.io/projected/b8f925a3-e179-41e3-af2e-1eb891af39be-kube-api-access-plrgd\") pod \"node-ca-gj5mm\" (UID: \"b8f925a3-e179-41e3-af2e-1eb891af39be\") " pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.200401 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.200371 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkm8\" (UniqueName: \"kubernetes.io/projected/ce3ad4a2-a07d-4f4e-b17f-387346234df0-kube-api-access-gtkm8\") pod \"ovnkube-node-x4jhr\" (UID: \"ce3ad4a2-a07d-4f4e-b17f-387346234df0\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.285535 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285437 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-kubelet\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.285535 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285491 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2bp\" (UniqueName: \"kubernetes.io/projected/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-kube-api-access-vw2bp\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.285766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285620 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-kubelet\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.285766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285666 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b9d180-ba25-43e8-9ddf-7a91608ea469-tmp-dir\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.285766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285698 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.285766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285723 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfsw\" (UniqueName: \"kubernetes.io/projected/f7b9d180-ba25-43e8-9ddf-7a91608ea469-kube-api-access-qhfsw\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.285766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285764 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-system-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285788 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-agent-certs\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285810 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-host\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285862 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-cnibin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285888 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-cni-binary-copy\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285917 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-socket-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285952 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.285977 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2j5k\" (UniqueName: \"kubernetes.io/projected/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-kube-api-access-z2j5k\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.286028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286002 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-sys\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286036 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286061 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-os-release\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286081 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-device-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286115 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-konnectivity-ca\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286136 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysconfig\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286157 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-systemd\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286203 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-conf-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286230 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-system-cni-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286258 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-registration-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286293 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cnibin\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286330 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286358 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxbp\" (UniqueName: \"kubernetes.io/projected/c7329761-9c29-4771-a1f6-561c797a93ce-kube-api-access-8jxbp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286391 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286384 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-etc-kubernetes\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286407 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286413 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7b9d180-ba25-43e8-9ddf-7a91608ea469-tmp-dir\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286432 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-etc-selinux\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286456 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286480 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-etc-tuned\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286505 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-tmp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286554 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-kubernetes\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286580 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-run\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286583 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286606 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhd68\" (UniqueName: \"kubernetes.io/projected/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-kube-api-access-xhd68\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286637 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-netns\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286667 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnllt\" (UniqueName: \"kubernetes.io/projected/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kube-api-access-bnllt\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286681 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-system-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286694 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-kubelet-config\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286721 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-host-slash\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286759 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-os-release\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.286965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286785 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-iptables-alerter-script\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286813 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-k8s-cni-cncf-io\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286841 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-daemon-config\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286887 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-multus-certs\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286911 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-dbus\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286936 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7b9d180-ba25-43e8-9ddf-7a91608ea469-hosts-file\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286969 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-modprobe-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287014 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-hostroot\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287043 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-multus\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287067 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq576\" (UniqueName: \"kubernetes.io/projected/d8b83ce7-2b96-4123-bae7-47993008f3b1-kube-api-access-lq576\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287099 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-sys-fs\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287124 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287152 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287186 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-var-lib-kubelet\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287236 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-socket-dir-parent\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287272 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287322 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-conf\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.287699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287351 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-lib-modules\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287388 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-bin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287412 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-host\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287488 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-cnibin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287512 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-bin\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287564 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-conf-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287608 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-system-cni-dir\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287683 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-registration-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287728 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cnibin\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.287892 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288116 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-cni-binary-copy\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288262 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-socket-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288267 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-sys\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288283 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-etc-kubernetes\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288352 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kubelet-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.288515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288471 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-cni-dir\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288545 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-os-release\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288470 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-etc-selinux\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.288960 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysconfig\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289036 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-dbus\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289053 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-konnectivity-ca\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289087 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-kubernetes\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289135 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-run\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.289264 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.289213 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.289289 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:13.789267196 +0000 UTC m=+3.078082618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289407 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-netns\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.286638 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-systemd\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289499 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6532cecb-fc83-485e-958e-f98fccb16ae0-kubelet-config\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.289522 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289543 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-host-slash\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.289572 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:13.789555625 +0000 UTC m=+3.078371039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289592 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-socket-dir-parent\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289602 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-os-release\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.289654 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289622 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7b9d180-ba25-43e8-9ddf-7a91608ea469-hosts-file\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289673 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-hostroot\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289676 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-var-lib-kubelet\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289517 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-binary-copy\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289746 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-sys-fs\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.289862 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-var-lib-cni-multus\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.290233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290162 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-iptables-alerter-script\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290247 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-multus-certs\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290411 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-sysctl-conf\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290426 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8b83ce7-2b96-4123-bae7-47993008f3b1-multus-daemon-config\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290426 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290499 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8b83ce7-2b96-4123-bae7-47993008f3b1-host-run-k8s-cni-cncf-io\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.290592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290535 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-etc-modprobe-d\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.290877 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290601 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7329761-9c29-4771-a1f6-561c797a93ce-lib-modules\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.290877 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290600 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/164cd44b-e796-4b1b-90c1-134b2cc3ba08-device-dir\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.290975 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290908 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.290975 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.290924 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41520bf2-b9f5-44e2-8c5b-0f497e79fa13-agent-certs\") pod \"konnectivity-agent-rlzmg\" (UID: \"41520bf2-b9f5-44e2-8c5b-0f497e79fa13\") " pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.292529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.292499 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-tmp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.293685 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.293657 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7329761-9c29-4771-a1f6-561c797a93ce-etc-tuned\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.303810 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.303738 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnllt\" (UniqueName: \"kubernetes.io/projected/164cd44b-e796-4b1b-90c1-134b2cc3ba08-kube-api-access-bnllt\") pod \"aws-ebs-csi-driver-node-znrmr\" (UID: \"164cd44b-e796-4b1b-90c1-134b2cc3ba08\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.304706 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.304671 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq576\" (UniqueName: \"kubernetes.io/projected/d8b83ce7-2b96-4123-bae7-47993008f3b1-kube-api-access-lq576\") pod \"multus-nbfwm\" (UID: \"d8b83ce7-2b96-4123-bae7-47993008f3b1\") " pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.305036 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.305010 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxbp\" (UniqueName: \"kubernetes.io/projected/c7329761-9c29-4771-a1f6-561c797a93ce-kube-api-access-8jxbp\") pod \"tuned-h8xk2\" (UID: \"c7329761-9c29-4771-a1f6-561c797a93ce\") " pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.305173 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.305140 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2j5k\" (UniqueName: \"kubernetes.io/projected/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-kube-api-access-z2j5k\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.305310 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.305222 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhd68\" (UniqueName: \"kubernetes.io/projected/ea64b3c4-ff10-43e7-afba-c8179c4b63b9-kube-api-access-xhd68\") pod \"iptables-alerter-xzv2p\" (UID: \"ea64b3c4-ff10-43e7-afba-c8179c4b63b9\") " pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.306833 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.306775 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfsw\" (UniqueName: \"kubernetes.io/projected/f7b9d180-ba25-43e8-9ddf-7a91608ea469-kube-api-access-qhfsw\") pod \"node-resolver-zqc8n\" (UID: \"f7b9d180-ba25-43e8-9ddf-7a91608ea469\") " pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.307244 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.307188 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2bp\" (UniqueName: \"kubernetes.io/projected/3b47ef9e-9cbf-423d-b6e7-b50c9102cf91-kube-api-access-vw2bp\") pod \"multus-additional-cni-plugins-djqxk\" (UID: \"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91\") " pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.323097 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.322989 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" event={"ID":"290d0c9e2524a1d4503c0280421e2430","Type":"ContainerStarted","Data":"b88568ccda8bae6a18e12db34fead541093ed144291d52e23ac545b40b106e26"} Jul 15 23:30:13.325122 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.324740 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" event={"ID":"21088ddecb53e5296158fe55990a64c7","Type":"ContainerStarted","Data":"306074174d1112433b2a2190b336c0fa76afdb79412edc827ce3e1f8029d2796"} Jul 15 23:30:13.374594 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.374557 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gj5mm" Jul 15 23:30:13.386099 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.386063 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:13.387137 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.387087 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f925a3_e179_41e3_af2e_1eb891af39be.slice/crio-610085e483db8fc2b1663fec17266eb72134b6ff54dfa7daf7e0e21e3f7b85d4 WatchSource:0}: Error finding container 610085e483db8fc2b1663fec17266eb72134b6ff54dfa7daf7e0e21e3f7b85d4: Status 404 returned error can't find the container with id 610085e483db8fc2b1663fec17266eb72134b6ff54dfa7daf7e0e21e3f7b85d4 Jul 15 23:30:13.398240 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.398205 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3ad4a2_a07d_4f4e_b17f_387346234df0.slice/crio-7ddbfb995d7918a13365b3c4412d9235b091a854b8bfa5b1bb67dd4d79ce7243 WatchSource:0}: Error finding container 7ddbfb995d7918a13365b3c4412d9235b091a854b8bfa5b1bb67dd4d79ce7243: Status 404 returned error can't find the container with id 7ddbfb995d7918a13365b3c4412d9235b091a854b8bfa5b1bb67dd4d79ce7243 Jul 15 23:30:13.400038 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.400003 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:13.407673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.407639 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nbfwm" Jul 15 23:30:13.411536 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.411493 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41520bf2_b9f5_44e2_8c5b_0f497e79fa13.slice/crio-4606735c556ce89b98f3795b5f4f1e0569cd73919ae40af036ece4ddace67f93 WatchSource:0}: Error finding container 4606735c556ce89b98f3795b5f4f1e0569cd73919ae40af036ece4ddace67f93: Status 404 returned error can't find the container with id 4606735c556ce89b98f3795b5f4f1e0569cd73919ae40af036ece4ddace67f93 Jul 15 23:30:13.419433 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.419391 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqc8n" Jul 15 23:30:13.421579 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.421394 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b83ce7_2b96_4123_bae7_47993008f3b1.slice/crio-57670601f00f06e8e4db64a5633f46dd9de27e40d3ac82ea40f890ac2ad86079 WatchSource:0}: Error finding container 57670601f00f06e8e4db64a5633f46dd9de27e40d3ac82ea40f890ac2ad86079: Status 404 returned error can't find the container with id 57670601f00f06e8e4db64a5633f46dd9de27e40d3ac82ea40f890ac2ad86079 Jul 15 23:30:13.428770 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.428744 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" Jul 15 23:30:13.432860 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.432829 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b9d180_ba25_43e8_9ddf_7a91608ea469.slice/crio-7fb0419c084fc059c07e4a94f56856cdb380cceb274df2a96d54460dc58c6416 WatchSource:0}: Error finding container 7fb0419c084fc059c07e4a94f56856cdb380cceb274df2a96d54460dc58c6416: Status 404 returned error can't find the container with id 7fb0419c084fc059c07e4a94f56856cdb380cceb274df2a96d54460dc58c6416 Jul 15 23:30:13.439706 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.439678 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" Jul 15 23:30:13.448409 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.448373 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djqxk" Jul 15 23:30:13.458655 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.458623 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzv2p" Jul 15 23:30:13.791370 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.791334 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:13.791514 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.791395 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:13.791514 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:13.791429 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:13.791596 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791524 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:13.791596 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791543 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:13.791596 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791582 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:14.791567739 +0000 UTC m=+4.080383147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:13.791747 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791607 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:14.791592635 +0000 UTC m=+4.080408047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:13.791747 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791542 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:13.791747 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791635 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:13.791747 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791651 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:13.791747 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:13.791693 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:14.791678606 +0000 UTC m=+4.080494019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:13.795585 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:13.795553 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7329761_9c29_4771_a1f6_561c797a93ce.slice/crio-c39b80195940a3628cc5725116ff508e131e5479b39fd07cafb7bc3bbfe7950a WatchSource:0}: Error finding container c39b80195940a3628cc5725116ff508e131e5479b39fd07cafb7bc3bbfe7950a: Status 404 returned error can't find the container with id c39b80195940a3628cc5725116ff508e131e5479b39fd07cafb7bc3bbfe7950a Jul 15 23:30:14.133782 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.133669 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 23:25:13 +0000 UTC" deadline="2028-03-17 09:36:57.390901339 +0000 UTC" Jul 15 23:30:14.133782 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.133723 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14650h6m43.257181771s" Jul 15 23:30:14.328850 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.328801 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"7ddbfb995d7918a13365b3c4412d9235b091a854b8bfa5b1bb67dd4d79ce7243"} Jul 15 23:30:14.330242 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.330208 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gj5mm" event={"ID":"b8f925a3-e179-41e3-af2e-1eb891af39be","Type":"ContainerStarted","Data":"610085e483db8fc2b1663fec17266eb72134b6ff54dfa7daf7e0e21e3f7b85d4"} Jul 15 23:30:14.331726 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.331693 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" event={"ID":"c7329761-9c29-4771-a1f6-561c797a93ce","Type":"ContainerStarted","Data":"c39b80195940a3628cc5725116ff508e131e5479b39fd07cafb7bc3bbfe7950a"} Jul 15 23:30:14.333415 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.333343 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqc8n" event={"ID":"f7b9d180-ba25-43e8-9ddf-7a91608ea469","Type":"ContainerStarted","Data":"7fb0419c084fc059c07e4a94f56856cdb380cceb274df2a96d54460dc58c6416"} Jul 15 23:30:14.334916 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.334868 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nbfwm" event={"ID":"d8b83ce7-2b96-4123-bae7-47993008f3b1","Type":"ContainerStarted","Data":"57670601f00f06e8e4db64a5633f46dd9de27e40d3ac82ea40f890ac2ad86079"} Jul 15 23:30:14.336197 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.336166 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlzmg" event={"ID":"41520bf2-b9f5-44e2-8c5b-0f497e79fa13","Type":"ContainerStarted","Data":"4606735c556ce89b98f3795b5f4f1e0569cd73919ae40af036ece4ddace67f93"} Jul 15 23:30:14.357179 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.357132 2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 23:30:14.604421 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:14.604387 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164cd44b_e796_4b1b_90c1_134b2cc3ba08.slice/crio-491fe1aee073f70a7a15d06ef3092e050b48993bd6c55c2dd6e3ae1e2e04a4f2 WatchSource:0}: Error finding container 491fe1aee073f70a7a15d06ef3092e050b48993bd6c55c2dd6e3ae1e2e04a4f2: Status 404 returned error can't find the container with id 491fe1aee073f70a7a15d06ef3092e050b48993bd6c55c2dd6e3ae1e2e04a4f2 Jul 15 23:30:14.605961 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:14.605927 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b47ef9e_9cbf_423d_b6e7_b50c9102cf91.slice/crio-3d1fb2fc49f7d78f10908160bf3a4131793a82f6968d61cbfb0c48c9e31759e4 WatchSource:0}: Error finding container 3d1fb2fc49f7d78f10908160bf3a4131793a82f6968d61cbfb0c48c9e31759e4: Status 404 returned error can't find the container with id 3d1fb2fc49f7d78f10908160bf3a4131793a82f6968d61cbfb0c48c9e31759e4 Jul 15 23:30:14.611198 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:14.611112 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea64b3c4_ff10_43e7_afba_c8179c4b63b9.slice/crio-6d34ded9d3c5d84df3fca1a21189147841a6e82186bfeac78e64d6f0c617c8f7 WatchSource:0}: Error finding container 6d34ded9d3c5d84df3fca1a21189147841a6e82186bfeac78e64d6f0c617c8f7: Status 404 returned error can't find the container with id 6d34ded9d3c5d84df3fca1a21189147841a6e82186bfeac78e64d6f0c617c8f7 Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.802618 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.802954 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:14.803004 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803183 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803221 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803236 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803302 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:16.803281396 +0000 UTC m=+6.092096808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803661 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803748 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:16.803715604 +0000 UTC m=+6.092531022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803752 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:14.803926 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:14.803807 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:16.80379037 +0000 UTC m=+6.092605786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:15.135973 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.135682 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 23:25:13 +0000 UTC" deadline="2028-02-29 10:06:25.961649127 +0000 UTC" Jul 15 23:30:15.135973 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.135725 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14242h36m10.825929321s" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.297307 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:15.297464 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.297925 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:15.298023 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.298094 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:15.298262 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:15.298172 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:15.350571 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.349811 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" event={"ID":"290d0c9e2524a1d4503c0280421e2430","Type":"ContainerStarted","Data":"a43895e20f90ecba49cd238b5389bd253ab0efe6861a0fb80beb865d8fd80ca5"} Jul 15 23:30:15.354232 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.353584 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" event={"ID":"21088ddecb53e5296158fe55990a64c7","Type":"ContainerStarted","Data":"ee070c392993dd5bf06ad5d210bd1953de74edc72674aab1d1f25931ee0fe819"} Jul 15 23:30:15.359960 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.359878 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerStarted","Data":"3d1fb2fc49f7d78f10908160bf3a4131793a82f6968d61cbfb0c48c9e31759e4"} Jul 15 23:30:15.362659 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.362618 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" event={"ID":"164cd44b-e796-4b1b-90c1-134b2cc3ba08","Type":"ContainerStarted","Data":"491fe1aee073f70a7a15d06ef3092e050b48993bd6c55c2dd6e3ae1e2e04a4f2"} Jul 15 23:30:15.364857 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:15.364822 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzv2p" event={"ID":"ea64b3c4-ff10-43e7-afba-c8179c4b63b9","Type":"ContainerStarted","Data":"6d34ded9d3c5d84df3fca1a21189147841a6e82186bfeac78e64d6f0c617c8f7"} Jul 15 23:30:16.402014 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.401930 2551 generic.go:356] "Generic (PLEG): container finished" podID="290d0c9e2524a1d4503c0280421e2430" containerID="a43895e20f90ecba49cd238b5389bd253ab0efe6861a0fb80beb865d8fd80ca5" exitCode=0 Jul 15 23:30:16.402462 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.402049 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" event={"ID":"290d0c9e2524a1d4503c0280421e2430","Type":"ContainerDied","Data":"a43895e20f90ecba49cd238b5389bd253ab0efe6861a0fb80beb865d8fd80ca5"} Jul 15 23:30:16.424171 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.422638 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-149.ec2.internal" podStartSLOduration=4.422611644 podStartE2EDuration="4.422611644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:15.385972897 +0000 UTC m=+4.674788329" watchObservedRunningTime="2026-07-15 23:30:16.422611644 +0000 UTC m=+5.711427081" Jul 15 23:30:16.826223 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.826100 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:16.826223 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.826197 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:16.826480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:16.826241 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:16.826480 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.826400 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:16.826480 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.826420 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:16.826480 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.826434 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:16.826666 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.826498 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:20.826478739 +0000 UTC m=+10.115294148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:16.826977 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.826951 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:16.827079 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.827013 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:20.826996336 +0000 UTC m=+10.115811767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:16.827079 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.827075 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:16.827227 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:16.827112 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:20.827099884 +0000 UTC m=+10.115915294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:17.298365 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:17.298256 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:17.298537 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:17.298391 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:17.298537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:17.298256 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:17.298537 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:17.298489 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:17.298537 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:17.298510 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:17.298730 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:17.298588 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:17.409114 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:17.409074 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlzmg" event={"ID":"41520bf2-b9f5-44e2-8c5b-0f497e79fa13","Type":"ContainerStarted","Data":"ffdc4ad6f9f8f21eedf01c2aae87f5bb6ea894ab29bbf80cb8ffb352b2fb17e7"} Jul 15 23:30:17.508559 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:17.508519 2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlzmg_41520bf2-b9f5-44e2-8c5b-0f497e79fa13/konnectivity-agent/0.log" limit=1024 Jul 15 23:30:19.298336 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:19.298251 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:19.298756 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:19.298385 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:19.298816 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:19.298771 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:19.298941 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:19.298865 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:19.299077 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:19.299028 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:19.299200 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:19.299158 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:20.864547 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:20.864506 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:20.865015 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.864696 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:20.865015 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.864777 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:28.864753931 +0000 UTC m=+18.153569340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:20.865240 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:20.865091 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:20.865240 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:20.865163 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:20.865240 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865219 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:20.865395 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865272 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:28.865256154 +0000 UTC m=+18.154071567 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:20.865395 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865292 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:20.865395 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865312 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:20.865395 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865323 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:20.865395 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:20.865370 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:28.865357732 +0000 UTC m=+18.154173141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:21.057697 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.057657 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:21.058421 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.058399 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:21.074397 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.074334 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rlzmg" podStartSLOduration=7.254159574 podStartE2EDuration="10.074310969s" totalImagesPullingTime="2.820151395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:17.425707638 +0000 UTC m=+6.714523069" watchObservedRunningTime="2026-07-15 23:30:21.074310969 +0000 UTC m=+10.363126399" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.298763 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:21.298877 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.298994 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:21.299081 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:21.299148 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:21.299260 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:21.299231 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:23.297405 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:23.297367 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:23.297860 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:23.297504 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:23.297937 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:23.297906 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:23.298025 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:23.297999 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:23.298185 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:23.298059 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:23.298185 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:23.298163 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:25.296856 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:25.296818 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:25.297334 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:25.296942 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:25.297334 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:25.297058 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:25.297334 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:25.297083 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:25.297334 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:25.297198 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:25.297334 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:25.297288 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:26.890615 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:26.890581 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:26.891087 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:26.890720 2551 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 15 23:30:26.891229 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:26.891206 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rlzmg" Jul 15 23:30:27.297405 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:27.297317 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:27.297568 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:27.297317 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:27.297568 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:27.297450 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:27.297675 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:27.297317 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:27.297730 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:27.297642 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:27.297730 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:27.297699 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:28.929423 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:28.929354 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:28.929423 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:28.929415 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:28.929466 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929510 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929538 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929558 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929571 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929584 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929588 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:44.929572662 +0000 UTC m=+34.218388069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929631 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:44.929615601 +0000 UTC m=+34.218431019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:28.930023 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:28.929649 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:30:44.929641931 +0000 UTC m=+34.218457338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:29.297517 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:29.297424 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:29.297687 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:29.297424 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:29.297687 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:29.297589 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:29.297687 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:29.297427 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:29.297687 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:29.297614 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:29.297869 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:29.297723 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:31.297797 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:31.297765 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:31.298218 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:31.297880 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:31.298218 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:31.297912 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:31.298218 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:31.297993 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:31.298218 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:31.298040 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:31.298218 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:31.298106 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:33.297378 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:33.297338 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:33.297763 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:33.297340 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:33.297763 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:33.297455 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:33.297763 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:33.297476 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:33.297763 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:33.297551 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:33.297763 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:33.297637 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:33.468472 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:33.468432 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" event={"ID":"290d0c9e2524a1d4503c0280421e2430","Type":"ContainerStarted","Data":"acce9fada998e16d09c43479b78159042cef8d4a35dcb3853e2d3d07ad34e604"} Jul 15 23:30:34.472979 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.472759 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"1e928ffa8386c06bcbb758bc8ea34b8e614ed49285aaeb03f8ab9bb1b9d36318"} Jul 15 23:30:34.473804 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.473099 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"821d1bd95b2fb755e5e4d70a2658706b37cfd58120f68ff70c4c30071f7bbdd5"} Jul 15 23:30:34.473804 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.473178 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"ffb3b7690fcf87ecffeb25ac830c4718e1c718c9d9424b12200f43d28c5d4d5e"} Jul 15 23:30:34.473804 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.473230 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"1c96bb37fedb58a6a12900e32144e374a3513aaf358a24c394f531a666cbd57e"} Jul 15 23:30:34.474381 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.474354 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gj5mm" event={"ID":"b8f925a3-e179-41e3-af2e-1eb891af39be","Type":"ContainerStarted","Data":"1d26f040f7b45037e0cfeca9720570052c66471d5421ba95baf0f4257f609c1c"} Jul 15 23:30:34.475852 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.475815 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" event={"ID":"c7329761-9c29-4771-a1f6-561c797a93ce","Type":"ContainerStarted","Data":"43124259bedece7046b9978d3f5e079d1b67e4a47b37b35e478b9e79f1f86201"} Jul 15 23:30:34.477438 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.477412 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqc8n" event={"ID":"f7b9d180-ba25-43e8-9ddf-7a91608ea469","Type":"ContainerStarted","Data":"425cdcb9ac6c7b9d7d74032633956f835ddd0a33722eadbad5de022d7d2a57f1"} Jul 15 23:30:34.478831 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.478810 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="970fef5b932c7e17b123be611bea97a3ffbb54b978a6e5f1f2c5536ddabbf991" exitCode=0 Jul 15 23:30:34.478958 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.478896 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"970fef5b932c7e17b123be611bea97a3ffbb54b978a6e5f1f2c5536ddabbf991"} Jul 15 23:30:34.480331 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.480306 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" event={"ID":"164cd44b-e796-4b1b-90c1-134b2cc3ba08","Type":"ContainerStarted","Data":"4c26c680f8346068a0975611c098f9b5ccd7cde2c32822dc9e950c275d8d64af"} Jul 15 23:30:34.481722 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.481702 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nbfwm" event={"ID":"d8b83ce7-2b96-4123-bae7-47993008f3b1","Type":"ContainerStarted","Data":"5e9590fd53a08c2213079a6bfe708f5cee00b11504d1de08b5d518aa58ae6e0b"} Jul 15 23:30:34.488550 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.488488 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gj5mm" podStartSLOduration=11.704950281 podStartE2EDuration="23.488468564s" totalImagesPullingTime="11.783518283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:34.488347002 +0000 UTC m=+23.777162432" watchObservedRunningTime="2026-07-15 23:30:34.488468564 +0000 UTC m=+23.777283993" Jul 15 23:30:34.504317 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.504264 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8xk2" podStartSLOduration=2.909787471 podStartE2EDuration="22.50424792s" totalImagesPullingTime="19.594460449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:34.503682891 +0000 UTC m=+23.792498322" watchObservedRunningTime="2026-07-15 23:30:34.50424792 +0000 UTC m=+23.793063349" Jul 15 23:30:34.542998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.542947 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-149.ec2.internal" podStartSLOduration=22.54292264 podStartE2EDuration="22.54292264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:34.542768899 +0000 UTC m=+23.831584327" watchObservedRunningTime="2026-07-15 23:30:34.54292264 +0000 UTC m=+23.831738069" Jul 15 23:30:34.588768 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.588715 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zqc8n" podStartSLOduration=2.63396846 podStartE2EDuration="22.588698378s" totalImagesPullingTime="19.954729918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:34.588699054 +0000 UTC m=+23.877514482" watchObservedRunningTime="2026-07-15 23:30:34.588698378 +0000 UTC m=+23.877513808" Jul 15 23:30:34.588905 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.588792 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nbfwm" podStartSLOduration=3.598210232 podStartE2EDuration="23.588787047s" totalImagesPullingTime="19.990576815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:34.56446677 +0000 UTC m=+23.853282199" watchObservedRunningTime="2026-07-15 23:30:34.588787047 +0000 UTC m=+23.877602476" Jul 15 23:30:34.921053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:34.921016 2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 23:30:35.156564 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.156408 2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T23:30:34.921047332Z","UUID":"717a3b8c-f8cd-4671-88de-0f8b93782a39","Handler":null,"Name":"","Endpoint":""} Jul 15 23:30:35.158298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.158168 2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 23:30:35.158298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.158219 2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 23:30:35.296855 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.296813 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:35.297144 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.296813 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:35.297144 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:35.296956 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:35.297144 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:35.297039 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:35.297144 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.296813 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:35.297144 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:35.297138 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:35.487466 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.487382 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" event={"ID":"164cd44b-e796-4b1b-90c1-134b2cc3ba08","Type":"ContainerStarted","Data":"19f3a47176fc25ec29c350c07d44f08dca2dbc925030d76b9dec0a3cc7c2e00c"} Jul 15 23:30:35.491150 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.491104 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"004adfc8b8e9a30a969e5adab1d24d776d096a18b1dbd0e19de83f568baa911e"} Jul 15 23:30:35.491318 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.491203 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"b94db3f8af21b1acf14afdd25f3f5292c9069ba920ebcaf41078df377e6630a6"} Jul 15 23:30:35.492869 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.492823 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzv2p" event={"ID":"ea64b3c4-ff10-43e7-afba-c8179c4b63b9","Type":"ContainerStarted","Data":"babf7681aef82c4d0ee2d65f75134d66bb3aa7540127d1aae8162acad5bee5df"} Jul 15 23:30:35.509701 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:35.509632 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzv2p" podStartSLOduration=4.732918848 podStartE2EDuration="23.509614252s" totalImagesPullingTime="18.776695404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:35.509442278 +0000 UTC m=+24.798257706" watchObservedRunningTime="2026-07-15 23:30:35.509614252 +0000 UTC m=+24.798429680" Jul 15 23:30:36.497766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:36.497679 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" event={"ID":"164cd44b-e796-4b1b-90c1-134b2cc3ba08","Type":"ContainerStarted","Data":"a8495971e5a0d86d3d1206bd635bfe5b90755e1b18549d8aef05690d204a851f"} Jul 15 23:30:36.517648 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:36.517580 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-znrmr" podStartSLOduration=3.4678493599999998 podStartE2EDuration="24.517556189s" totalImagesPullingTime="21.049706829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:36.516974902 +0000 UTC m=+25.805790332" watchObservedRunningTime="2026-07-15 23:30:36.517556189 +0000 UTC m=+25.806371643" Jul 15 23:30:37.297362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:37.297323 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:37.297562 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:37.297378 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:37.297562 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:37.297446 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:37.297562 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:37.297464 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:37.297562 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:37.297530 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:37.297804 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:37.297615 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:37.504382 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:37.504093 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"2ef5a9d7d5212e4dbda27983338ab3ab2aa56a87509e58ee04a4cea1fee7affe"} Jul 15 23:30:39.297597 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.297301 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:39.298338 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.297299 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:39.298338 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:39.297727 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:39.298338 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.297347 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:39.298338 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:39.297761 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:39.298338 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:39.297827 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:39.510224 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.510191 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="f614156e3c979fa3f24486a1b3e8e92b8a92404d272ec7d8e571629283c72e5a" exitCode=0 Jul 15 23:30:39.510421 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.510283 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"f614156e3c979fa3f24486a1b3e8e92b8a92404d272ec7d8e571629283c72e5a"} Jul 15 23:30:39.514112 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.514083 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" event={"ID":"ce3ad4a2-a07d-4f4e-b17f-387346234df0","Type":"ContainerStarted","Data":"240855686a114885414fc4ab862aec6b845c34d07c7637753f3d7046cd0c2834"} Jul 15 23:30:39.514379 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.514356 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:39.514495 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.514433 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:39.514495 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.514475 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:39.531350 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.531320 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:39.531506 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.531423 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:30:39.562632 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:39.562536 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" podStartSLOduration=8.233427212 podStartE2EDuration="28.562515962s" totalImagesPullingTime="20.32908875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:39.561248909 +0000 UTC m=+28.850064339" watchObservedRunningTime="2026-07-15 23:30:39.562515962 +0000 UTC m=+28.851331393" Jul 15 23:30:40.518948 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:40.518903 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerStarted","Data":"7f6cf308d3857f7c3039893866ab750b5089a552505365ca9f9c834737aeb604"} Jul 15 23:30:41.297728 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.297687 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:41.297899 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.297790 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:41.297899 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.297806 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:41.297899 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.297828 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:41.298000 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.297930 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:41.298043 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.298021 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:41.326608 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.326565 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km96l"] Jul 15 23:30:41.329838 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.329808 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5ftd"] Jul 15 23:30:41.330425 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.330403 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mggsj"] Jul 15 23:30:41.523234 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.523199 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="7f6cf308d3857f7c3039893866ab750b5089a552505365ca9f9c834737aeb604" exitCode=0 Jul 15 23:30:41.523680 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.523295 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"7f6cf308d3857f7c3039893866ab750b5089a552505365ca9f9c834737aeb604"} Jul 15 23:30:41.523680 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.523343 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:41.523680 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.523453 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:41.523927 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.523909 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:41.524026 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.524005 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:41.524080 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:41.524037 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:41.524175 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:41.524155 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:43.297380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:43.297101 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:43.297813 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:43.297115 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:43.297813 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:43.297412 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:43.297813 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:43.297147 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:43.297813 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:43.297471 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:43.297813 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:43.297575 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:43.530201 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:43.530165 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="de6fc6465adee2d3d71869e57280b4ad44f8b8250a1f52983cad2f91775cb824" exitCode=0 Jul 15 23:30:43.530398 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:43.530243 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"de6fc6465adee2d3d71869e57280b4ad44f8b8250a1f52983cad2f91775cb824"} Jul 15 23:30:44.948628 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:44.948589 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:44.948699 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:44.948739 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948774 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948857 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948859 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs podName:f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98 nodeName:}" failed. No retries permitted until 2026-07-15 23:31:16.948837031 +0000 UTC m=+66.237652444 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs") pod "network-metrics-daemon-mggsj" (UID: "f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948934 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret podName:6532cecb-fc83-485e-958e-f98fccb16ae0 nodeName:}" failed. No retries permitted until 2026-07-15 23:31:16.948917807 +0000 UTC m=+66.237733215 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret") pod "global-pull-secret-syncer-x5ftd" (UID: "6532cecb-fc83-485e-958e-f98fccb16ae0") : object "kube-system"/"original-pull-secret" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948862 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948957 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948969 2551 projected.go:196] Error preparing data for projected volume kube-api-access-x6fxb for pod openshift-network-diagnostics/network-check-target-km96l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:44.949349 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:44.948999 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb podName:0bca1717-764f-4b24-adab-94cc74e47ce0 nodeName:}" failed. No retries permitted until 2026-07-15 23:31:16.948990544 +0000 UTC m=+66.237805999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x6fxb" (UniqueName: "kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb") pod "network-check-target-km96l" (UID: "0bca1717-764f-4b24-adab-94cc74e47ce0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 23:30:45.297011 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:45.296977 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:45.297204 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:45.296977 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:45.297204 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:45.297118 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:45.297204 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:45.297194 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:45.297204 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:45.296977 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:45.297412 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:45.297269 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:47.297316 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.297278 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:47.297884 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.297279 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:47.297884 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:47.297413 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x5ftd" podUID="6532cecb-fc83-485e-958e-f98fccb16ae0" Jul 15 23:30:47.297884 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.297285 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:47.297884 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:47.297504 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mggsj" podUID="f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98" Jul 15 23:30:47.297884 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:47.297570 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km96l" podUID="0bca1717-764f-4b24-adab-94cc74e47ce0" Jul 15 23:30:47.563892 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.563863 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-149.ec2.internal" event="NodeReady" Jul 15 23:30:47.564036 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.564023 2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 23:30:47.610517 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.610483 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j294m"] Jul 15 23:30:47.612558 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.612531 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mt28x"] Jul 15 23:30:47.612713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.612658 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.614607 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.614584 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.616284 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.616263 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 23:30:47.616394 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.616312 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 23:30:47.616907 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.616889 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hnnk9\"" Jul 15 23:30:47.617185 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.617170 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 23:30:47.617279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.617192 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 23:30:47.618017 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.617999 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 23:30:47.618107 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.618001 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8tb2c\"" Jul 15 23:30:47.632905 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.632873 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt28x"] Jul 15 23:30:47.633809 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.633783 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j294m"] Jul 15 23:30:47.773441 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773401 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638213a2-5e50-4ea6-a930-1b4eea3f7058-cert\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.773550 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773447 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q246\" (UniqueName: \"kubernetes.io/projected/638213a2-5e50-4ea6-a930-1b4eea3f7058-kube-api-access-2q246\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.773550 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773480 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.773550 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773537 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386eada3-b78e-4126-9a67-89fc6029278b-tmp-dir\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.773658 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773553 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7pz\" (UniqueName: \"kubernetes.io/projected/386eada3-b78e-4126-9a67-89fc6029278b-kube-api-access-kx7pz\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.773658 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.773589 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386eada3-b78e-4126-9a67-89fc6029278b-config-volume\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.874842 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874744 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638213a2-5e50-4ea6-a930-1b4eea3f7058-cert\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.874842 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874787 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q246\" (UniqueName: \"kubernetes.io/projected/638213a2-5e50-4ea6-a930-1b4eea3f7058-kube-api-access-2q246\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.874842 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874814 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.874842 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874845 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386eada3-b78e-4126-9a67-89fc6029278b-tmp-dir\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.875198 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874860 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7pz\" (UniqueName: \"kubernetes.io/projected/386eada3-b78e-4126-9a67-89fc6029278b-kube-api-access-kx7pz\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.875198 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.874884 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386eada3-b78e-4126-9a67-89fc6029278b-config-volume\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.875198 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:47.875012 2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 23:30:47.875198 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:47.875165 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls podName:386eada3-b78e-4126-9a67-89fc6029278b nodeName:}" failed. No retries permitted until 2026-07-15 23:30:48.375116389 +0000 UTC m=+37.663931822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls") pod "dns-default-j294m" (UID: "386eada3-b78e-4126-9a67-89fc6029278b") : secret "dns-default-metrics-tls" not found Jul 15 23:30:47.875386 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.875242 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/386eada3-b78e-4126-9a67-89fc6029278b-tmp-dir\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.875483 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.875463 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/386eada3-b78e-4126-9a67-89fc6029278b-config-volume\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.879319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.879294 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/638213a2-5e50-4ea6-a930-1b4eea3f7058-cert\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.883382 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.883348 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7pz\" (UniqueName: \"kubernetes.io/projected/386eada3-b78e-4126-9a67-89fc6029278b-kube-api-access-kx7pz\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:47.883577 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.883549 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q246\" (UniqueName: \"kubernetes.io/projected/638213a2-5e50-4ea6-a930-1b4eea3f7058-kube-api-access-2q246\") pod \"ingress-canary-mt28x\" (UID: \"638213a2-5e50-4ea6-a930-1b4eea3f7058\") " pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:47.931717 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:47.931680 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mt28x" Jul 15 23:30:48.071462 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.071275 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mt28x"] Jul 15 23:30:48.074606 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:48.074571 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638213a2_5e50_4ea6_a930_1b4eea3f7058.slice/crio-d92c326dbe3050ed7d8537459aee989e86e5c6f8ec02bbd7dbf7d898e4248516 WatchSource:0}: Error finding container d92c326dbe3050ed7d8537459aee989e86e5c6f8ec02bbd7dbf7d898e4248516: Status 404 returned error can't find the container with id d92c326dbe3050ed7d8537459aee989e86e5c6f8ec02bbd7dbf7d898e4248516 Jul 15 23:30:48.192029 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.191925 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jhft7"] Jul 15 23:30:48.195283 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.195142 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.198414 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.198387 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 23:30:48.198569 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.198492 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 23:30:48.198569 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.198550 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 23:30:48.198688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.198571 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-25gj9\"" Jul 15 23:30:48.199720 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.199704 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 23:30:48.202867 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.202844 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jhft7"] Jul 15 23:30:48.278600 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.278551 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv992\" (UniqueName: \"kubernetes.io/projected/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-kube-api-access-vv992\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.278780 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.278621 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-key\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.278780 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.278645 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.379477 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.379424 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv992\" (UniqueName: \"kubernetes.io/projected/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-kube-api-access-vv992\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.379477 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.379480 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:48.379477 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.379498 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-key\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.380029 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.379520 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.380029 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:48.379588 2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 23:30:48.380029 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:48.379660 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls podName:386eada3-b78e-4126-9a67-89fc6029278b nodeName:}" failed. No retries permitted until 2026-07-15 23:30:49.379642449 +0000 UTC m=+38.668457856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls") pod "dns-default-j294m" (UID: "386eada3-b78e-4126-9a67-89fc6029278b") : secret "dns-default-metrics-tls" not found Jul 15 23:30:48.380312 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.380293 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-cabundle\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.382825 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.382802 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-signing-key\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.387616 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.387582 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv992\" (UniqueName: \"kubernetes.io/projected/6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb-kube-api-access-vv992\") pod \"service-ca-79f68b9f4f-jhft7\" (UID: \"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb\") " pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.506019 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.505989 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" Jul 15 23:30:48.546002 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.545965 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="4bd536bd855b993776668d0deee8ba4cb766bb35d280bdf3533a1d54bcd0447c" exitCode=0 Jul 15 23:30:48.546193 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.546012 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"4bd536bd855b993776668d0deee8ba4cb766bb35d280bdf3533a1d54bcd0447c"} Jul 15 23:30:48.547641 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.547617 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt28x" event={"ID":"638213a2-5e50-4ea6-a930-1b4eea3f7058","Type":"ContainerStarted","Data":"d92c326dbe3050ed7d8537459aee989e86e5c6f8ec02bbd7dbf7d898e4248516"} Jul 15 23:30:48.666610 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:48.666525 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-jhft7"] Jul 15 23:30:48.671405 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:30:48.671208 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad7c96f_ba39_4af5_b1d4_bcce50d41fdb.slice/crio-635ebb08593d875d63a049148a6e492e3b758b72b3a94bd4203e55e6ddeb799d WatchSource:0}: Error finding container 635ebb08593d875d63a049148a6e492e3b758b72b3a94bd4203e55e6ddeb799d: Status 404 returned error can't find the container with id 635ebb08593d875d63a049148a6e492e3b758b72b3a94bd4203e55e6ddeb799d Jul 15 23:30:49.297053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.297018 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:30:49.297274 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.297183 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:30:49.297567 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.297183 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:30:49.306404 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.306374 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 23:30:49.306572 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.306548 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 23:30:49.308588 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.307701 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjrwg\"" Jul 15 23:30:49.308588 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.308006 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ctqbg\"" Jul 15 23:30:49.308588 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.308240 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 23:30:49.308588 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.308445 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 23:30:49.389573 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.389527 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:49.390071 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:49.389699 2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 23:30:49.390071 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:49.389783 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls podName:386eada3-b78e-4126-9a67-89fc6029278b nodeName:}" failed. No retries permitted until 2026-07-15 23:30:51.389761972 +0000 UTC m=+40.678577379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls") pod "dns-default-j294m" (UID: "386eada3-b78e-4126-9a67-89fc6029278b") : secret "dns-default-metrics-tls" not found Jul 15 23:30:49.551900 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.551744 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" event={"ID":"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb","Type":"ContainerStarted","Data":"635ebb08593d875d63a049148a6e492e3b758b72b3a94bd4203e55e6ddeb799d"} Jul 15 23:30:49.556049 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.556001 2551 generic.go:356] "Generic (PLEG): container finished" podID="3b47ef9e-9cbf-423d-b6e7-b50c9102cf91" containerID="2f656fca594e036ef95838aa184a9f4a356946cf4b2c415466646e780fc6e65b" exitCode=0 Jul 15 23:30:49.556225 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:49.556060 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerDied","Data":"2f656fca594e036ef95838aa184a9f4a356946cf4b2c415466646e780fc6e65b"} Jul 15 23:30:51.039985 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.039814 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqc8n_f7b9d180-ba25-43e8-9ddf-7a91608ea469/dns-node-resolver/0.log" Jul 15 23:30:51.407473 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.407433 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:51.407682 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:51.407553 2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 23:30:51.407682 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:51.407607 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls podName:386eada3-b78e-4126-9a67-89fc6029278b nodeName:}" failed. No retries permitted until 2026-07-15 23:30:55.407592365 +0000 UTC m=+44.696407771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls") pod "dns-default-j294m" (UID: "386eada3-b78e-4126-9a67-89fc6029278b") : secret "dns-default-metrics-tls" not found Jul 15 23:30:51.562142 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.562097 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mt28x" event={"ID":"638213a2-5e50-4ea6-a930-1b4eea3f7058","Type":"ContainerStarted","Data":"a0a010edd80105059ac9469ae82488e193f5040e9da0aae985ae7c477c28aeec"} Jul 15 23:30:51.563630 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.563596 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" event={"ID":"6ad7c96f-ba39-4af5-b1d4-bcce50d41fdb","Type":"ContainerStarted","Data":"4a8e14333d37165a8d365268fa965b0f761346f92f483811ecdf4cb4041989f0"} Jul 15 23:30:51.566753 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.566725 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djqxk" event={"ID":"3b47ef9e-9cbf-423d-b6e7-b50c9102cf91","Type":"ContainerStarted","Data":"a3c526bf3288e8d0efc5715c34441f213c1497201a05d961071e0b801c70f10b"} Jul 15 23:30:51.580376 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.580311 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mt28x" podStartSLOduration=1.750626682 podStartE2EDuration="4.580295054s" totalImagesPullingTime="2.829668372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:51.578953209 +0000 UTC m=+40.867768615" watchObservedRunningTime="2026-07-15 23:30:51.580295054 +0000 UTC m=+40.869110482" Jul 15 23:30:51.594072 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.594017 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-jhft7" podStartSLOduration=1.358307967 podStartE2EDuration="3.593999434s" totalImagesPullingTime="2.235691467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:51.593384433 +0000 UTC m=+40.882199859" watchObservedRunningTime="2026-07-15 23:30:51.593999434 +0000 UTC m=+40.882814891" Jul 15 23:30:51.614249 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.614192 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-djqxk" podStartSLOduration=10.532118791 podStartE2EDuration="39.614173198s" totalImagesPullingTime="29.082054407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:30:51.612335672 +0000 UTC m=+40.901151100" watchObservedRunningTime="2026-07-15 23:30:51.614173198 +0000 UTC m=+40.902988626" Jul 15 23:30:51.839332 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:51.839301 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gj5mm_b8f925a3-e179-41e3-af2e-1eb891af39be/node-ca/0.log" Jul 15 23:30:52.438204 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:52.438172 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mt28x_638213a2-5e50-4ea6-a930-1b4eea3f7058/serve-healthcheck-canary/0.log" Jul 15 23:30:55.436930 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:30:55.436882 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:30:55.437371 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:55.437048 2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 23:30:55.437371 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:30:55.437112 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls podName:386eada3-b78e-4126-9a67-89fc6029278b nodeName:}" failed. No retries permitted until 2026-07-15 23:31:03.437095718 +0000 UTC m=+52.725911134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls") pod "dns-default-j294m" (UID: "386eada3-b78e-4126-9a67-89fc6029278b") : secret "dns-default-metrics-tls" not found Jul 15 23:31:03.496074 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:03.496025 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:31:03.499615 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:03.499580 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/386eada3-b78e-4126-9a67-89fc6029278b-metrics-tls\") pod \"dns-default-j294m\" (UID: \"386eada3-b78e-4126-9a67-89fc6029278b\") " pod="openshift-dns/dns-default-j294m" Jul 15 23:31:03.525621 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:03.525581 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j294m" Jul 15 23:31:03.708516 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:03.708483 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j294m"] Jul 15 23:31:03.712228 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:03.712200 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386eada3_b78e_4126_9a67_89fc6029278b.slice/crio-ad3f0a057ffc36333b31100ef8f88102822595834e8dbb458e7b7a29d3158973 WatchSource:0}: Error finding container ad3f0a057ffc36333b31100ef8f88102822595834e8dbb458e7b7a29d3158973: Status 404 returned error can't find the container with id ad3f0a057ffc36333b31100ef8f88102822595834e8dbb458e7b7a29d3158973 Jul 15 23:31:04.599091 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:04.599050 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j294m" event={"ID":"386eada3-b78e-4126-9a67-89fc6029278b","Type":"ContainerStarted","Data":"ad3f0a057ffc36333b31100ef8f88102822595834e8dbb458e7b7a29d3158973"} Jul 15 23:31:06.605996 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:06.605954 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j294m" event={"ID":"386eada3-b78e-4126-9a67-89fc6029278b","Type":"ContainerStarted","Data":"62f85c80580785f5849fa676ffaf29ffd721dced74d621fd532d348c37250bc6"} Jul 15 23:31:06.606396 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:06.606057 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j294m" event={"ID":"386eada3-b78e-4126-9a67-89fc6029278b","Type":"ContainerStarted","Data":"23b5b856792883b7e42c5ad0e19f6f073296889d265a66daf42d258f1f8b7d5f"} Jul 15 23:31:06.606396 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:06.606097 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j294m" Jul 15 23:31:06.625645 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:06.625593 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j294m" podStartSLOduration=17.104695539 podStartE2EDuration="19.625577692s" totalImagesPullingTime="2.520882153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:06.625090038 +0000 UTC m=+55.913905479" watchObservedRunningTime="2026-07-15 23:31:06.625577692 +0000 UTC m=+55.914393120" Jul 15 23:31:07.428905 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.428871 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8m2p6"] Jul 15 23:31:07.430764 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.430745 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.434028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.433995 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 23:31:07.434213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.434034 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 23:31:07.434213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.434151 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 23:31:07.435151 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.435116 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 23:31:07.435266 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.435162 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xzgrt\"" Jul 15 23:31:07.450729 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.450685 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m2p6"] Jul 15 23:31:07.486290 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.486257 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg"] Jul 15 23:31:07.488238 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.488217 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld"] Jul 15 23:31:07.488381 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.488359 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.490666 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.490637 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.494362 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.494329 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 23:31:07.494872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.494855 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 23:31:07.495085 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.495064 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 23:31:07.495351 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.495242 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 23:31:07.495947 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.495927 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 23:31:07.496119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.496104 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 23:31:07.496438 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.496422 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 23:31:07.496438 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.496426 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 23:31:07.518231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.518194 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg"] Jul 15 23:31:07.524770 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524737 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524776 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsnb\" (UniqueName: \"kubernetes.io/projected/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-api-access-pfsnb\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524803 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524826 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524875 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-data-volume\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524897 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpc6\" (UniqueName: \"kubernetes.io/projected/711094e1-6367-44c3-bac2-5ea540c2a511-kube-api-access-6lpc6\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524933 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-crio-socket\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.524956 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524947 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3934f48f-2914-4b1c-ad36-9c4c3d045a26-tmp\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524966 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996qm\" (UniqueName: \"kubernetes.io/projected/3934f48f-2914-4b1c-ad36-9c4c3d045a26-kube-api-access-996qm\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.524986 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.525005 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.525034 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3934f48f-2914-4b1c-ad36-9c4c3d045a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.525080 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.525239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.525103 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/711094e1-6367-44c3-bac2-5ea540c2a511-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.554714 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.554684 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld"] Jul 15 23:31:07.625743 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625695 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3934f48f-2914-4b1c-ad36-9c4c3d045a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.625743 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625745 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625766 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/711094e1-6367-44c3-bac2-5ea540c2a511-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625802 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625819 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsnb\" (UniqueName: \"kubernetes.io/projected/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-api-access-pfsnb\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625836 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625850 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.625869 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-data-volume\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626002 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpc6\" (UniqueName: \"kubernetes.io/projected/711094e1-6367-44c3-bac2-5ea540c2a511-kube-api-access-6lpc6\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626064 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-crio-socket\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626090 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3934f48f-2914-4b1c-ad36-9c4c3d045a26-tmp\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626119 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996qm\" (UniqueName: \"kubernetes.io/projected/3934f48f-2914-4b1c-ad36-9c4c3d045a26-kube-api-access-996qm\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626169 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626211 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.626936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626576 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3934f48f-2914-4b1c-ad36-9c4c3d045a26-tmp\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.626936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626611 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626655 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-crio-socket\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.626936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.626847 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-data-volume\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.627747 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.627724 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/711094e1-6367-44c3-bac2-5ea540c2a511-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.628831 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.628803 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.629152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.628899 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.629152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.629058 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3934f48f-2914-4b1c-ad36-9c4c3d045a26-klusterlet-config\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.629249 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.629181 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-ca\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.629333 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.629318 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.629750 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.629726 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/711094e1-6367-44c3-bac2-5ea540c2a511-hub\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.640114 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.640075 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsnb\" (UniqueName: \"kubernetes.io/projected/aae883d1-fd1b-4c3b-8a85-3417ff6e93f7-kube-api-access-pfsnb\") pod \"insights-runtime-extractor-8m2p6\" (UID: \"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7\") " pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.641106 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.641041 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpc6\" (UniqueName: \"kubernetes.io/projected/711094e1-6367-44c3-bac2-5ea540c2a511-kube-api-access-6lpc6\") pod \"cluster-proxy-proxy-agent-b76998bb8-lffld\" (UID: \"711094e1-6367-44c3-bac2-5ea540c2a511\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.641440 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.641415 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-996qm\" (UniqueName: \"kubernetes.io/projected/3934f48f-2914-4b1c-ad36-9c4c3d045a26-kube-api-access-996qm\") pod \"klusterlet-addon-workmgr-69558f9949-bg4hg\" (UID: \"3934f48f-2914-4b1c-ad36-9c4c3d045a26\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.674433 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.674394 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-545dbd4fc7-pb425"] Jul 15 23:31:07.677351 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.677323 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.680645 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.680572 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 23:31:07.681334 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.681309 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mj9fx\"" Jul 15 23:31:07.681445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.681334 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 23:31:07.681868 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.681847 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 23:31:07.690665 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.690632 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 23:31:07.691393 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.691370 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545dbd4fc7-pb425"] Jul 15 23:31:07.727402 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727359 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-registry-tls\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727440 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-image-registry-private-configuration\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727461 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-bound-sa-token\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727481 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxrb\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-kube-api-access-xcxrb\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727499 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-installation-pull-secrets\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727516 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-trusted-ca\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727783 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727582 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d303399-f322-4d8b-8555-514663a4740f-ca-trust-extracted\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.727783 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.727641 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-registry-certificates\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.740476 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.740442 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m2p6" Jul 15 23:31:07.800257 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.800226 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:07.814769 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.814725 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828667 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-installation-pull-secrets\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828708 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-trusted-ca\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828730 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d303399-f322-4d8b-8555-514663a4740f-ca-trust-extracted\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828782 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-registry-certificates\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828809 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-registry-tls\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828890 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-image-registry-private-configuration\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828916 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-bound-sa-token\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.829348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.828951 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxrb\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-kube-api-access-xcxrb\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.830540 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.830482 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-trusted-ca\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.831055 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.830983 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d303399-f322-4d8b-8555-514663a4740f-registry-certificates\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.831192 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.831140 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d303399-f322-4d8b-8555-514663a4740f-ca-trust-extracted\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.832899 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.832869 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-installation-pull-secrets\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.833718 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.833656 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-registry-tls\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.833718 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.833659 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d303399-f322-4d8b-8555-514663a4740f-image-registry-private-configuration\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.842313 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.842268 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-bound-sa-token\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.842541 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.842513 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxrb\" (UniqueName: \"kubernetes.io/projected/6d303399-f322-4d8b-8555-514663a4740f-kube-api-access-xcxrb\") pod \"image-registry-545dbd4fc7-pb425\" (UID: \"6d303399-f322-4d8b-8555-514663a4740f\") " pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:07.892308 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.892216 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m2p6"] Jul 15 23:31:07.896268 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:07.896225 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae883d1_fd1b_4c3b_8a85_3417ff6e93f7.slice/crio-f43d7b9f651e69666fbe94625e5baa389727663f004f3aa63d24e05cf13833ff WatchSource:0}: Error finding container f43d7b9f651e69666fbe94625e5baa389727663f004f3aa63d24e05cf13833ff: Status 404 returned error can't find the container with id f43d7b9f651e69666fbe94625e5baa389727663f004f3aa63d24e05cf13833ff Jul 15 23:31:07.952584 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.952491 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg"] Jul 15 23:31:07.956016 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:07.955989 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3934f48f_2914_4b1c_ad36_9c4c3d045a26.slice/crio-6584f12d8f64d8a8cc8180c5b259de2761bd6aac4187c12a36f31e269159348d WatchSource:0}: Error finding container 6584f12d8f64d8a8cc8180c5b259de2761bd6aac4187c12a36f31e269159348d: Status 404 returned error can't find the container with id 6584f12d8f64d8a8cc8180c5b259de2761bd6aac4187c12a36f31e269159348d Jul 15 23:31:07.969952 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.969920 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld"] Jul 15 23:31:07.973418 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:07.973388 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711094e1_6367_44c3_bac2_5ea540c2a511.slice/crio-4f3af5b44607a9f2d4ed790224634ca62866da71ac8aa4542edf0bd5fcd42314 WatchSource:0}: Error finding container 4f3af5b44607a9f2d4ed790224634ca62866da71ac8aa4542edf0bd5fcd42314: Status 404 returned error can't find the container with id 4f3af5b44607a9f2d4ed790224634ca62866da71ac8aa4542edf0bd5fcd42314 Jul 15 23:31:07.991713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:07.991667 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:08.126601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.126570 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545dbd4fc7-pb425"] Jul 15 23:31:08.131226 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:08.131197 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d303399_f322_4d8b_8555_514663a4740f.slice/crio-fa585e7fa56b4edfc04afe7f2c4b3246730dae86e4587a1e7efc2afee626f609 WatchSource:0}: Error finding container fa585e7fa56b4edfc04afe7f2c4b3246730dae86e4587a1e7efc2afee626f609: Status 404 returned error can't find the container with id fa585e7fa56b4edfc04afe7f2c4b3246730dae86e4587a1e7efc2afee626f609 Jul 15 23:31:08.617024 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.616924 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m2p6" event={"ID":"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7","Type":"ContainerStarted","Data":"f43d7b9f651e69666fbe94625e5baa389727663f004f3aa63d24e05cf13833ff"} Jul 15 23:31:08.619962 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.619900 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" event={"ID":"711094e1-6367-44c3-bac2-5ea540c2a511","Type":"ContainerStarted","Data":"4f3af5b44607a9f2d4ed790224634ca62866da71ac8aa4542edf0bd5fcd42314"} Jul 15 23:31:08.623293 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.623256 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" event={"ID":"6d303399-f322-4d8b-8555-514663a4740f","Type":"ContainerStarted","Data":"614979400ce701bb0b203f97ffc228afdbb842e937f16aa16de24103db47d9dc"} Jul 15 23:31:08.623467 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.623321 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" event={"ID":"6d303399-f322-4d8b-8555-514663a4740f","Type":"ContainerStarted","Data":"fa585e7fa56b4edfc04afe7f2c4b3246730dae86e4587a1e7efc2afee626f609"} Jul 15 23:31:08.624092 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.624035 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:08.627155 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.627066 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" event={"ID":"3934f48f-2914-4b1c-ad36-9c4c3d045a26","Type":"ContainerStarted","Data":"6584f12d8f64d8a8cc8180c5b259de2761bd6aac4187c12a36f31e269159348d"} Jul 15 23:31:08.645973 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:08.645041 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" podStartSLOduration=1.6450161890000001 podStartE2EDuration="1.645016189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:08.643381374 +0000 UTC m=+57.932196806" watchObservedRunningTime="2026-07-15 23:31:08.645016189 +0000 UTC m=+57.933831622" Jul 15 23:31:11.534507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:11.534475 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4jhr" Jul 15 23:31:15.655971 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:15.655781 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" event={"ID":"711094e1-6367-44c3-bac2-5ea540c2a511","Type":"ContainerStarted","Data":"4f381fe2c47181aad241e638aff54a1c7d94dcbbde9ca5498d22788108707ecf"} Jul 15 23:31:15.655971 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:15.655841 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" event={"ID":"711094e1-6367-44c3-bac2-5ea540c2a511","Type":"ContainerStarted","Data":"5d7d509c77dd0f89a4aab8149398b400cc9197d648ce2842de4165a9bb28358e"} Jul 15 23:31:15.655971 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:15.655859 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" event={"ID":"711094e1-6367-44c3-bac2-5ea540c2a511","Type":"ContainerStarted","Data":"5fb13b9137877105e3804eec00952b7ca9af998a2d1362b4555b99f9bbfcc41f"} Jul 15 23:31:15.658348 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:15.658305 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m2p6" event={"ID":"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7","Type":"ContainerStarted","Data":"cd08370321a96ec74f0465bf46972d9bdc4bb76ddece0fdfe0e8959d52214eec"} Jul 15 23:31:15.677492 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:15.677043 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b76998bb8-lffld" podStartSLOduration=1.7401412760000001 podStartE2EDuration="8.677017732s" totalImagesPullingTime="6.936876456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:15.675085415 +0000 UTC m=+64.963900844" watchObservedRunningTime="2026-07-15 23:31:15.677017732 +0000 UTC m=+64.965833162" Jul 15 23:31:16.611988 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:16.611958 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j294m" Jul 15 23:31:17.017870 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.017830 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:31:17.018363 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.017932 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:31:17.018363 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.017968 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:31:17.020952 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.020923 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 23:31:17.022496 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.022461 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 23:31:17.022642 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.022519 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 23:31:17.030798 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.030767 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98-metrics-certs\") pod \"network-metrics-daemon-mggsj\" (UID: \"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98\") " pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:31:17.030958 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.030854 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6532cecb-fc83-485e-958e-f98fccb16ae0-original-pull-secret\") pod \"global-pull-secret-syncer-x5ftd\" (UID: \"6532cecb-fc83-485e-958e-f98fccb16ae0\") " pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:31:17.031035 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.030968 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 23:31:17.041549 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.041514 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fxb\" (UniqueName: \"kubernetes.io/projected/0bca1717-764f-4b24-adab-94cc74e47ce0-kube-api-access-x6fxb\") pod \"network-check-target-km96l\" (UID: \"0bca1717-764f-4b24-adab-94cc74e47ce0\") " pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:31:17.218017 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.217978 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ctqbg\"" Jul 15 23:31:17.225426 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.225388 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:31:17.229431 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.229405 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wjrwg\"" Jul 15 23:31:17.234724 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.234693 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5ftd" Jul 15 23:31:17.237484 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:17.237456 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mggsj" Jul 15 23:31:19.457232 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.457120 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km96l"] Jul 15 23:31:19.477945 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.477909 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mggsj"] Jul 15 23:31:19.483024 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.482994 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5ftd"] Jul 15 23:31:19.522627 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:19.522590 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bca1717_764f_4b24_adab_94cc74e47ce0.slice/crio-d389c4dffd2c0e4f8f6fe70d7659b3baaf25db3ce089fbb2e522c1ee29ff387d WatchSource:0}: Error finding container d389c4dffd2c0e4f8f6fe70d7659b3baaf25db3ce089fbb2e522c1ee29ff387d: Status 404 returned error can't find the container with id d389c4dffd2c0e4f8f6fe70d7659b3baaf25db3ce089fbb2e522c1ee29ff387d Jul 15 23:31:19.677524 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.677457 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5ftd" event={"ID":"6532cecb-fc83-485e-958e-f98fccb16ae0","Type":"ContainerStarted","Data":"b6cfa5a196ca9c7941079419df50609194c7594bbebcbbc91432f5f93a236a83"} Jul 15 23:31:19.679149 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.679108 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" event={"ID":"3934f48f-2914-4b1c-ad36-9c4c3d045a26","Type":"ContainerStarted","Data":"53b07132ce8952903959cc9ff39a0c72f4ddcfa8bce7ec6a2acc2042d80d6390"} Jul 15 23:31:19.680272 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.680250 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km96l" event={"ID":"0bca1717-764f-4b24-adab-94cc74e47ce0","Type":"ContainerStarted","Data":"d389c4dffd2c0e4f8f6fe70d7659b3baaf25db3ce089fbb2e522c1ee29ff387d"} Jul 15 23:31:19.681981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.681944 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m2p6" event={"ID":"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7","Type":"ContainerStarted","Data":"7b8366861e2635fbe82d8e70f27d022e7640bad32267de01cd7c429ff4e67c17"} Jul 15 23:31:19.683222 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.683188 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mggsj" event={"ID":"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98","Type":"ContainerStarted","Data":"f773180d1179e86402265ec306d18b872da7234d280a494ca484ad6cb1bada32"} Jul 15 23:31:19.697056 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:19.696999 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" podStartSLOduration=1.358225345 podStartE2EDuration="12.696979677s" totalImagesPullingTime="11.338754332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:19.696343985 +0000 UTC m=+68.985159416" watchObservedRunningTime="2026-07-15 23:31:19.696979677 +0000 UTC m=+68.985795100" Jul 15 23:31:20.688189 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:20.688093 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:20.690470 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:20.690235 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69558f9949-bg4hg" Jul 15 23:31:21.697062 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:21.696513 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mggsj" event={"ID":"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98","Type":"ContainerStarted","Data":"5c85a415d1e1122d31de12a7679c774f0c53198547deb98e9388cc71c5417216"} Jul 15 23:31:21.697062 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:21.696575 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mggsj" event={"ID":"f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98","Type":"ContainerStarted","Data":"d92a36599ba71aafdbfa7fc4b25975d600197720cec1bbaad5672d1d2ae44630"} Jul 15 23:31:21.714177 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:21.713856 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mggsj" podStartSLOduration=69.228128288 podStartE2EDuration="1m10.713831923s" totalImagesPullingTime="1.485703635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:21.711595509 +0000 UTC m=+71.000410938" watchObservedRunningTime="2026-07-15 23:31:21.713831923 +0000 UTC m=+71.002647354" Jul 15 23:31:26.717771 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.717729 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5ftd" event={"ID":"6532cecb-fc83-485e-958e-f98fccb16ae0","Type":"ContainerStarted","Data":"ae1c58ca8abdcd74537a96d9c1d33aa5ed0c996418d3d422add7869a4eb35ef4"} Jul 15 23:31:26.719446 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.719411 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km96l" event={"ID":"0bca1717-764f-4b24-adab-94cc74e47ce0","Type":"ContainerStarted","Data":"82a8c9dfb35c38d888a56b8db77745c816b7e54409169095ac3b574cf6a5f606"} Jul 15 23:31:26.719582 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.719544 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:31:26.721772 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.721748 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m2p6" event={"ID":"aae883d1-fd1b-4c3b-8a85-3417ff6e93f7","Type":"ContainerStarted","Data":"dfb360d2d7a903fd44179b4d6be36abcf66fa863dbb5a4bb156f05101faf03c9"} Jul 15 23:31:26.734778 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.734722 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x5ftd" podStartSLOduration=68.08971773 podStartE2EDuration="1m14.73470438s" totalImagesPullingTime="6.64498665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:26.73326403 +0000 UTC m=+76.022079460" watchObservedRunningTime="2026-07-15 23:31:26.73470438 +0000 UTC m=+76.023519800" Jul 15 23:31:26.751293 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.751230 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-km96l" podStartSLOduration=69.121260607 podStartE2EDuration="1m15.751206775s" totalImagesPullingTime="6.629946168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:26.750852806 +0000 UTC m=+76.039668268" watchObservedRunningTime="2026-07-15 23:31:26.751206775 +0000 UTC m=+76.040022205" Jul 15 23:31:26.770786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:26.770727 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8m2p6" podStartSLOduration=1.684817899 podStartE2EDuration="19.770708519s" totalImagesPullingTime="18.08589062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 23:31:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:26.769943833 +0000 UTC m=+76.058759269" watchObservedRunningTime="2026-07-15 23:31:26.770708519 +0000 UTC m=+76.059523948" Jul 15 23:31:30.639604 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:30.639572 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-545dbd4fc7-pb425" Jul 15 23:31:36.165758 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.165615 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cbbb9"] Jul 15 23:31:36.168737 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.168705 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.171691 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.171653 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 23:31:36.171873 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.171853 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 23:31:36.171977 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.171963 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 23:31:36.172084 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.172066 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 23:31:36.172195 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.172181 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 23:31:36.172859 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.172840 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 23:31:36.172953 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.172883 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lqmk\"" Jul 15 23:31:36.265865 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.265821 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-sys\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266057 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.265874 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-wtmp\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266057 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.265902 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266057 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.265935 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-metrics-client-ca\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.266090 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hphms\" (UniqueName: \"kubernetes.io/projected/09d68cb9-c524-481e-912f-2a14ed2d9327-kube-api-access-hphms\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.266161 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-tls\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.266197 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-textfile\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.266225 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-root\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.266414 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.266314 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.366904 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.366862 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hphms\" (UniqueName: \"kubernetes.io/projected/09d68cb9-c524-481e-912f-2a14ed2d9327-kube-api-access-hphms\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.366919 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-tls\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.366950 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-textfile\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.366977 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-root\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367056 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367105 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-sys\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367143 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-wtmp\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367169 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367060 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-root\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367242 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-sys\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367287 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-metrics-client-ca\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367316 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-wtmp\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367652 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367549 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-textfile\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367896 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367876 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-accelerators-collector-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.367992 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.367936 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09d68cb9-c524-481e-912f-2a14ed2d9327-metrics-client-ca\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.369505 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.369472 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-tls\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.369661 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.369640 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09d68cb9-c524-481e-912f-2a14ed2d9327-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.377830 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.377792 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hphms\" (UniqueName: \"kubernetes.io/projected/09d68cb9-c524-481e-912f-2a14ed2d9327-kube-api-access-hphms\") pod \"node-exporter-cbbb9\" (UID: \"09d68cb9-c524-481e-912f-2a14ed2d9327\") " pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.478922 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.478828 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cbbb9" Jul 15 23:31:36.492219 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:31:36.492172 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d68cb9_c524_481e_912f_2a14ed2d9327.slice/crio-0cc0bb7623fe291d9bc3333d1a4db756747053a102a1132daec6ff1b908dcd40 WatchSource:0}: Error finding container 0cc0bb7623fe291d9bc3333d1a4db756747053a102a1132daec6ff1b908dcd40: Status 404 returned error can't find the container with id 0cc0bb7623fe291d9bc3333d1a4db756747053a102a1132daec6ff1b908dcd40 Jul 15 23:31:36.756957 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:36.756922 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbbb9" event={"ID":"09d68cb9-c524-481e-912f-2a14ed2d9327","Type":"ContainerStarted","Data":"0cc0bb7623fe291d9bc3333d1a4db756747053a102a1132daec6ff1b908dcd40"} Jul 15 23:31:37.762918 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:37.762882 2551 generic.go:356] "Generic (PLEG): container finished" podID="09d68cb9-c524-481e-912f-2a14ed2d9327" containerID="97e9328901e62ad00897ade0cc6f4b143bea3f9b83a6a008e2b022c226ec78d6" exitCode=0 Jul 15 23:31:37.763329 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:37.762928 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbbb9" event={"ID":"09d68cb9-c524-481e-912f-2a14ed2d9327","Type":"ContainerDied","Data":"97e9328901e62ad00897ade0cc6f4b143bea3f9b83a6a008e2b022c226ec78d6"} Jul 15 23:31:38.769437 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:38.769399 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbbb9" event={"ID":"09d68cb9-c524-481e-912f-2a14ed2d9327","Type":"ContainerStarted","Data":"cd8820aa3bda633cdbce6e670e78a61f4cab1f178dfac073340579104945f04f"} Jul 15 23:31:38.769827 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:38.769454 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cbbb9" event={"ID":"09d68cb9-c524-481e-912f-2a14ed2d9327","Type":"ContainerStarted","Data":"2560b9ec93184bb82481928285a0d5fc96990bc2c761edf2e5afab77676fd56f"} Jul 15 23:31:38.793658 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:38.791882 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cbbb9" podStartSLOduration=1.927992682 podStartE2EDuration="2.79185543s" totalImagesPullingTime="795.399497ms" totalInitContainerRuntime="68.463251ms" isStatefulPod=true podCreationTimestamp="2026-07-15 23:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:31:38.788837044 +0000 UTC m=+88.077652482" watchObservedRunningTime="2026-07-15 23:31:38.79185543 +0000 UTC m=+88.080670863" Jul 15 23:31:57.727265 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:31:57.727236 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-km96l" Jul 15 23:34:50.102316 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.102278 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk"] Jul 15 23:34:50.104575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.104554 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.107294 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.107248 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 23:34:50.107294 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.107275 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lzp9k\"" Jul 15 23:34:50.108773 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.108749 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 23:34:50.113689 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.113667 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk"] Jul 15 23:34:50.145051 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.145016 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk585\" (UniqueName: \"kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.145051 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.145059 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.145304 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.145086 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.246436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.246394 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk585\" (UniqueName: \"kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.246633 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.246444 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.246633 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.246481 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.246882 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.246849 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.246950 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.246884 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.259233 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.259200 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk585\" (UniqueName: \"kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.414184 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.414077 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:34:50.544191 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:50.543990 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk"] Jul 15 23:34:50.546710 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:34:50.546683 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92c728f_58e4_42f3_8d23_3def966c300d.slice/crio-cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4 WatchSource:0}: Error finding container cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4: Status 404 returned error can't find the container with id cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4 Jul 15 23:34:51.407150 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:51.407094 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" event={"ID":"e92c728f-58e4-42f3-8d23-3def966c300d","Type":"ContainerStarted","Data":"cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4"} Jul 15 23:34:57.430451 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:57.430415 2551 generic.go:356] "Generic (PLEG): container finished" podID="e92c728f-58e4-42f3-8d23-3def966c300d" containerID="997eb9a7bef7bc3e8096ed9d6b287d041700b6b2b3da659a97f603c1246e2d33" exitCode=0 Jul 15 23:34:57.430836 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:34:57.430508 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" event={"ID":"e92c728f-58e4-42f3-8d23-3def966c300d","Type":"ContainerDied","Data":"997eb9a7bef7bc3e8096ed9d6b287d041700b6b2b3da659a97f603c1246e2d33"} Jul 15 23:35:00.442101 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:00.442067 2551 generic.go:356] "Generic (PLEG): container finished" podID="e92c728f-58e4-42f3-8d23-3def966c300d" containerID="a5674481a30aa43ededc9264106f8ad5a300171c827ffcb78d6b1e1152c55190" exitCode=0 Jul 15 23:35:00.442601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:00.442161 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" event={"ID":"e92c728f-58e4-42f3-8d23-3def966c300d","Type":"ContainerDied","Data":"a5674481a30aa43ededc9264106f8ad5a300171c827ffcb78d6b1e1152c55190"} Jul 15 23:35:07.470507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:07.470470 2551 generic.go:356] "Generic (PLEG): container finished" podID="e92c728f-58e4-42f3-8d23-3def966c300d" containerID="e2c1e2b34204b2a45b3d5f92e660a9055f9821770f82cd88b24db270214a536c" exitCode=0 Jul 15 23:35:07.470919 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:07.470515 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" event={"ID":"e92c728f-58e4-42f3-8d23-3def966c300d","Type":"ContainerDied","Data":"e2c1e2b34204b2a45b3d5f92e660a9055f9821770f82cd88b24db270214a536c"} Jul 15 23:35:08.608751 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.608720 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:35:08.697280 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.697245 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk585\" (UniqueName: \"kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585\") pod \"e92c728f-58e4-42f3-8d23-3def966c300d\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " Jul 15 23:35:08.697492 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.697318 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util\") pod \"e92c728f-58e4-42f3-8d23-3def966c300d\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " Jul 15 23:35:08.697492 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.697378 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle\") pod \"e92c728f-58e4-42f3-8d23-3def966c300d\" (UID: \"e92c728f-58e4-42f3-8d23-3def966c300d\") " Jul 15 23:35:08.697766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.697742 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle" (OuterVolumeSpecName: "bundle") pod "e92c728f-58e4-42f3-8d23-3def966c300d" (UID: "e92c728f-58e4-42f3-8d23-3def966c300d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:35:08.699510 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.699469 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585" (OuterVolumeSpecName: "kube-api-access-xk585") pod "e92c728f-58e4-42f3-8d23-3def966c300d" (UID: "e92c728f-58e4-42f3-8d23-3def966c300d"). InnerVolumeSpecName "kube-api-access-xk585". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:35:08.701888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.701858 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util" (OuterVolumeSpecName: "util") pod "e92c728f-58e4-42f3-8d23-3def966c300d" (UID: "e92c728f-58e4-42f3-8d23-3def966c300d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:35:08.798554 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.798495 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:08.798554 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.798543 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk585\" (UniqueName: \"kubernetes.io/projected/e92c728f-58e4-42f3-8d23-3def966c300d-kube-api-access-xk585\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:08.798554 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:08.798556 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e92c728f-58e4-42f3-8d23-3def966c300d-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:09.481659 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:09.481625 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" event={"ID":"e92c728f-58e4-42f3-8d23-3def966c300d","Type":"ContainerDied","Data":"cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4"} Jul 15 23:35:09.481659 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:09.481661 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwcshk" Jul 15 23:35:09.481862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:09.481673 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cabf784fa7385e71a2db5c6c18dbbc3bf78f6fb13d6a65fc8815359df97286d4" Jul 15 23:35:11.086601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:11.086576 2551 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 23:35:14.091158 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091109 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss"] Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091365 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="util" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091375 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="util" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091386 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="extract" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091391 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="extract" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091399 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="pull" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091406 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="pull" Jul 15 23:35:14.093436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.091465 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92c728f-58e4-42f3-8d23-3def966c300d" containerName="extract" Jul 15 23:35:14.094354 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.094336 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.097274 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.097244 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 23:35:14.098371 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.098348 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 23:35:14.098532 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.098377 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8pt7h\"" Jul 15 23:35:14.102851 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.102824 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss"] Jul 15 23:35:14.236325 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.236285 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38925f12-408a-40a7-be41-4fe5ff3907f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.236325 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.236322 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mpg\" (UniqueName: \"kubernetes.io/projected/38925f12-408a-40a7-be41-4fe5ff3907f0-kube-api-access-54mpg\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.337810 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.337755 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38925f12-408a-40a7-be41-4fe5ff3907f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.337810 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.337810 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mpg\" (UniqueName: \"kubernetes.io/projected/38925f12-408a-40a7-be41-4fe5ff3907f0-kube-api-access-54mpg\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.338306 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.338282 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38925f12-408a-40a7-be41-4fe5ff3907f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.346339 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.346277 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mpg\" (UniqueName: \"kubernetes.io/projected/38925f12-408a-40a7-be41-4fe5ff3907f0-kube-api-access-54mpg\") pod \"openshift-lws-operator-fd8ccff4c-fbxss\" (UID: \"38925f12-408a-40a7-be41-4fe5ff3907f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.404113 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.404076 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" Jul 15 23:35:14.538190 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.538164 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss"] Jul 15 23:35:14.540810 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:35:14.540783 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38925f12_408a_40a7_be41_4fe5ff3907f0.slice/crio-6999c1a33560940b837c6c55da8dc3c71da8c30a68bcff13c7b6a0c4450424bd WatchSource:0}: Error finding container 6999c1a33560940b837c6c55da8dc3c71da8c30a68bcff13c7b6a0c4450424bd: Status 404 returned error can't find the container with id 6999c1a33560940b837c6c55da8dc3c71da8c30a68bcff13c7b6a0c4450424bd Jul 15 23:35:14.542553 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:14.542532 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:35:15.505870 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:15.505816 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" event={"ID":"38925f12-408a-40a7-be41-4fe5ff3907f0","Type":"ContainerStarted","Data":"6999c1a33560940b837c6c55da8dc3c71da8c30a68bcff13c7b6a0c4450424bd"} Jul 15 23:35:16.510909 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:16.510880 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" event={"ID":"38925f12-408a-40a7-be41-4fe5ff3907f0","Type":"ContainerStarted","Data":"d7c08cd363c67dd176b42b0733dd43fe09410052b8b7c33ff7aa0c7ebeedb3e5"} Jul 15 23:35:16.532901 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:16.532825 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fbxss" podStartSLOduration=0.626308838 podStartE2EDuration="2.532805684s" totalImagesPullingTime="1.906496846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:35:16.530679764 +0000 UTC m=+305.819495192" watchObservedRunningTime="2026-07-15 23:35:16.532805684 +0000 UTC m=+305.821621113" Jul 15 23:35:18.784045 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.784004 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj"] Jul 15 23:35:18.787562 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.787534 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.790319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.790297 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 23:35:18.791647 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.791616 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 23:35:18.791647 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.791618 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lzp9k\"" Jul 15 23:35:18.794596 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.794555 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj"] Jul 15 23:35:18.876978 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.876944 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.877187 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.876990 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbzs\" (UniqueName: \"kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.877187 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.877045 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.978489 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.978452 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.978688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.978508 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbzs\" (UniqueName: \"kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.978688 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.978635 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.978886 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.978868 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.978963 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.978947 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:18.987276 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:18.987239 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbzs\" (UniqueName: \"kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:19.098082 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:19.097980 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:19.229387 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:19.229361 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj"] Jul 15 23:35:19.232005 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:35:19.231979 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742d7dd2_9a70_4b2b_8158_b1854883ceeb.slice/crio-0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031 WatchSource:0}: Error finding container 0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031: Status 404 returned error can't find the container with id 0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031 Jul 15 23:35:19.522931 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:19.522887 2551 generic.go:356] "Generic (PLEG): container finished" podID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerID="4525715581ce7b92b22833a773c0e8653acbbdfab6ca57825b9b89659efc7486" exitCode=0 Jul 15 23:35:19.523327 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:19.522966 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" event={"ID":"742d7dd2-9a70-4b2b-8158-b1854883ceeb","Type":"ContainerDied","Data":"4525715581ce7b92b22833a773c0e8653acbbdfab6ca57825b9b89659efc7486"} Jul 15 23:35:19.523327 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:19.523047 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" event={"ID":"742d7dd2-9a70-4b2b-8158-b1854883ceeb","Type":"ContainerStarted","Data":"0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031"} Jul 15 23:35:20.527860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:20.527826 2551 generic.go:356] "Generic (PLEG): container finished" podID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerID="9e0f575895a5e3dc48cb9ac1987a07574a0d54e104e44d866e5071bf459a6f55" exitCode=0 Jul 15 23:35:20.528271 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:20.527919 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" event={"ID":"742d7dd2-9a70-4b2b-8158-b1854883ceeb","Type":"ContainerDied","Data":"9e0f575895a5e3dc48cb9ac1987a07574a0d54e104e44d866e5071bf459a6f55"} Jul 15 23:35:21.533044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:21.533008 2551 generic.go:356] "Generic (PLEG): container finished" podID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerID="f1dd287ed65ac66b61c2dd1eca080ceb4f0d2f1ebe14151f728703ead8baac66" exitCode=0 Jul 15 23:35:21.533540 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:21.533091 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" event={"ID":"742d7dd2-9a70-4b2b-8158-b1854883ceeb","Type":"ContainerDied","Data":"f1dd287ed65ac66b61c2dd1eca080ceb4f0d2f1ebe14151f728703ead8baac66"} Jul 15 23:35:22.662643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.662616 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:22.807610 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.807522 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle\") pod \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " Jul 15 23:35:22.807610 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.807592 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util\") pod \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " Jul 15 23:35:22.807816 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.807640 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgbzs\" (UniqueName: \"kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs\") pod \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\" (UID: \"742d7dd2-9a70-4b2b-8158-b1854883ceeb\") " Jul 15 23:35:22.808251 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.808215 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle" (OuterVolumeSpecName: "bundle") pod "742d7dd2-9a70-4b2b-8158-b1854883ceeb" (UID: "742d7dd2-9a70-4b2b-8158-b1854883ceeb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:35:22.809722 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.809696 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs" (OuterVolumeSpecName: "kube-api-access-pgbzs") pod "742d7dd2-9a70-4b2b-8158-b1854883ceeb" (UID: "742d7dd2-9a70-4b2b-8158-b1854883ceeb"). InnerVolumeSpecName "kube-api-access-pgbzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:35:22.813513 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.813481 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util" (OuterVolumeSpecName: "util") pod "742d7dd2-9a70-4b2b-8158-b1854883ceeb" (UID: "742d7dd2-9a70-4b2b-8158-b1854883ceeb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:35:22.908335 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.908297 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:22.908335 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.908324 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/742d7dd2-9a70-4b2b-8158-b1854883ceeb-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:22.908335 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:22.908333 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbzs\" (UniqueName: \"kubernetes.io/projected/742d7dd2-9a70-4b2b-8158-b1854883ceeb-kube-api-access-pgbzs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:35:23.541812 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:23.541784 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" Jul 15 23:35:23.541976 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:23.541781 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af266pj" event={"ID":"742d7dd2-9a70-4b2b-8158-b1854883ceeb","Type":"ContainerDied","Data":"0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031"} Jul 15 23:35:23.541976 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:23.541901 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8b19c3c5d5a6730718c406b09e1a5975e9b8ad97821e342c1a49761fb5f031" Jul 15 23:35:33.693505 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693469 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw"] Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693697 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="pull" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693708 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="pull" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693717 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="extract" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693722 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="extract" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693730 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="util" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693736 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="util" Jul 15 23:35:33.693955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.693778 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="742d7dd2-9a70-4b2b-8158-b1854883ceeb" containerName="extract" Jul 15 23:35:33.704375 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.704347 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.708071 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.708042 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 23:35:33.708071 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.708065 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 23:35:33.708323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.708042 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 23:35:33.708323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.708097 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 23:35:33.708452 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.708439 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mnxnd\"" Jul 15 23:35:33.719549 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.719519 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw"] Jul 15 23:35:33.788308 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.788266 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-manifests\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.788488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.788323 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-tmp\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.788488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.788351 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.788488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.788384 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.788488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.788429 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmn9\" (UniqueName: \"kubernetes.io/projected/930c319f-8335-460c-80a2-fcd4d6cec804-kube-api-access-6rmn9\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.888847 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.888799 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-manifests\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.889044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.888877 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-tmp\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.889044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.888904 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.889044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.888949 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.889044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.888973 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmn9\" (UniqueName: \"kubernetes.io/projected/930c319f-8335-460c-80a2-fcd4d6cec804-kube-api-access-6rmn9\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.889942 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.889913 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-manifests\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.891620 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.891595 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/930c319f-8335-460c-80a2-fcd4d6cec804-tmp\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.892243 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.892221 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-webhook-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.892325 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.892244 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/930c319f-8335-460c-80a2-fcd4d6cec804-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:33.896903 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:33.896875 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmn9\" (UniqueName: \"kubernetes.io/projected/930c319f-8335-460c-80a2-fcd4d6cec804-kube-api-access-6rmn9\") pod \"opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw\" (UID: \"930c319f-8335-460c-80a2-fcd4d6cec804\") " pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:34.015491 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:34.015463 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:34.158461 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:34.158401 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw"] Jul 15 23:35:34.160863 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:35:34.160833 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930c319f_8335_460c_80a2_fcd4d6cec804.slice/crio-5e1166f26b5ee511e80d9448adb9de45a6e9b0c13c6a85eaffbabf9126fb2780 WatchSource:0}: Error finding container 5e1166f26b5ee511e80d9448adb9de45a6e9b0c13c6a85eaffbabf9126fb2780: Status 404 returned error can't find the container with id 5e1166f26b5ee511e80d9448adb9de45a6e9b0c13c6a85eaffbabf9126fb2780 Jul 15 23:35:34.580110 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:34.580076 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" event={"ID":"930c319f-8335-460c-80a2-fcd4d6cec804","Type":"ContainerStarted","Data":"5e1166f26b5ee511e80d9448adb9de45a6e9b0c13c6a85eaffbabf9126fb2780"} Jul 15 23:35:37.594582 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:37.594476 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" event={"ID":"930c319f-8335-460c-80a2-fcd4d6cec804","Type":"ContainerStarted","Data":"6d1aea0d608787db25e20e2cb6b0a40d4198423b6f7bc121074118999f87d53d"} Jul 15 23:35:38.599326 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:38.599291 2551 generic.go:356] "Generic (PLEG): container finished" podID="930c319f-8335-460c-80a2-fcd4d6cec804" containerID="6d1aea0d608787db25e20e2cb6b0a40d4198423b6f7bc121074118999f87d53d" exitCode=0 Jul 15 23:35:38.599708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:38.599352 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" event={"ID":"930c319f-8335-460c-80a2-fcd4d6cec804","Type":"ContainerDied","Data":"6d1aea0d608787db25e20e2cb6b0a40d4198423b6f7bc121074118999f87d53d"} Jul 15 23:35:39.605146 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:39.605096 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" event={"ID":"930c319f-8335-460c-80a2-fcd4d6cec804","Type":"ContainerStarted","Data":"cf47e0846bc72f0c8402db2c135b4f7a151f31867e8fa2c5ca95acf0e2fd318b"} Jul 15 23:35:39.605543 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:39.605339 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:39.628291 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:39.628240 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" podStartSLOduration=2.176082218 podStartE2EDuration="6.628222348s" totalImagesPullingTime="3.422729576s" totalInitContainerRuntime="1.029410554s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:35:39.625726926 +0000 UTC m=+328.914542380" watchObservedRunningTime="2026-07-15 23:35:39.628222348 +0000 UTC m=+328.917037851" Jul 15 23:35:50.610782 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:50.610744 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw" Jul 15 23:35:53.856644 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.856605 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b2mrj"] Jul 15 23:35:53.862149 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.862093 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t"] Jul 15 23:35:53.862319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.862289 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:53.864060 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.864039 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:53.866977 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.866957 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 23:35:53.867105 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.866981 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 23:35:53.867732 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.867713 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 23:35:53.867826 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.867787 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 23:35:53.867903 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.867717 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nptn2\"" Jul 15 23:35:53.868037 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.868019 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 23:35:53.869880 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.869861 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sdpm6\"" Jul 15 23:35:53.878518 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.878463 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b2mrj"] Jul 15 23:35:53.884032 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.884002 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t"] Jul 15 23:35:53.916602 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.916568 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn42\" (UniqueName: \"kubernetes.io/projected/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-kube-api-access-4tn42\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:53.916786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.916633 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tmp\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:53.916786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.916662 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:53.916786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.916683 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:53.916786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:53.916734 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skl8x\" (UniqueName: \"kubernetes.io/projected/671e0999-eb83-489f-908d-875529d5668e-kube-api-access-skl8x\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.017245 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.017207 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skl8x\" (UniqueName: \"kubernetes.io/projected/671e0999-eb83-489f-908d-875529d5668e-kube-api-access-skl8x\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.017420 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.017263 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn42\" (UniqueName: \"kubernetes.io/projected/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-kube-api-access-4tn42\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.017420 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.017318 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tmp\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.017499 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.017446 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.017499 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.017484 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.017657 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.017643 2551 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 15 23:35:54.017704 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.017651 2551 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 23:35:54.017741 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.017703 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs podName:0c14bfa3-4cfd-444b-a4a0-05c05dcc5989 nodeName:}" failed. No retries permitted until 2026-07-15 23:35:54.517684477 +0000 UTC m=+343.806499887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs") pod "kube-auth-proxy-7bddb6d9b-9bf4t" (UID: "0c14bfa3-4cfd-444b-a4a0-05c05dcc5989") : secret "kube-auth-proxy-tls" not found Jul 15 23:35:54.017803 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.017736 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert podName:671e0999-eb83-489f-908d-875529d5668e nodeName:}" failed. No retries permitted until 2026-07-15 23:35:54.517717164 +0000 UTC m=+343.806532578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert") pod "odh-model-controller-7998c67dc4-b2mrj" (UID: "671e0999-eb83-489f-908d-875529d5668e") : secret "odh-model-controller-webhook-cert" not found Jul 15 23:35:54.019871 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.019836 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tmp\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.027399 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.027366 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skl8x\" (UniqueName: \"kubernetes.io/projected/671e0999-eb83-489f-908d-875529d5668e-kube-api-access-skl8x\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.027399 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.027388 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn42\" (UniqueName: \"kubernetes.io/projected/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-kube-api-access-4tn42\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.520960 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.520929 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.520960 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.520964 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.523433 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.523399 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/671e0999-eb83-489f-908d-875529d5668e-cert\") pod \"odh-model-controller-7998c67dc4-b2mrj\" (UID: \"671e0999-eb83-489f-908d-875529d5668e\") " pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.523548 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.523513 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c14bfa3-4cfd-444b-a4a0-05c05dcc5989-tls-certs\") pod \"kube-auth-proxy-7bddb6d9b-9bf4t\" (UID: \"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989\") " pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.559279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.559244 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-78f9q"] Jul 15 23:35:54.561698 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.561683 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.564525 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.564496 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 23:35:54.574044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.574011 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-78f9q"] Jul 15 23:35:54.622194 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.622163 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7nnq\" (UniqueName: \"kubernetes.io/projected/1af7a9c9-f783-4f73-9b67-a869f4761843-kube-api-access-t7nnq\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.622376 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.622202 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.723318 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.723277 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7nnq\" (UniqueName: \"kubernetes.io/projected/1af7a9c9-f783-4f73-9b67-a869f4761843-kube-api-access-t7nnq\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.723318 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.723316 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.723552 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.723417 2551 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 15 23:35:54.723552 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:35:54.723480 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs podName:1af7a9c9-f783-4f73-9b67-a869f4761843 nodeName:}" failed. No retries permitted until 2026-07-15 23:35:55.22346322 +0000 UTC m=+344.512278628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs") pod "model-serving-api-54d89bd5fc-78f9q" (UID: "1af7a9c9-f783-4f73-9b67-a869f4761843") : secret "model-serving-api-tls" not found Jul 15 23:35:54.739652 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.739627 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7nnq\" (UniqueName: \"kubernetes.io/projected/1af7a9c9-f783-4f73-9b67-a869f4761843-kube-api-access-t7nnq\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:54.775608 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.775472 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:35:54.786579 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.786553 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" Jul 15 23:35:54.943605 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.943577 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-b2mrj"] Jul 15 23:35:54.945928 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:35:54.945900 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671e0999_eb83_489f_908d_875529d5668e.slice/crio-0928013bbfd52a4a33ba57d383712a6c9280e155a70a2925f7ff516b4aadf6bc WatchSource:0}: Error finding container 0928013bbfd52a4a33ba57d383712a6c9280e155a70a2925f7ff516b4aadf6bc: Status 404 returned error can't find the container with id 0928013bbfd52a4a33ba57d383712a6c9280e155a70a2925f7ff516b4aadf6bc Jul 15 23:35:54.964679 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:54.964637 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t"] Jul 15 23:35:54.969010 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:35:54.968978 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c14bfa3_4cfd_444b_a4a0_05c05dcc5989.slice/crio-622c53a599c055a022a4dbe2f4b1b1790cfec293c66d2265867c9061e40cf6d4 WatchSource:0}: Error finding container 622c53a599c055a022a4dbe2f4b1b1790cfec293c66d2265867c9061e40cf6d4: Status 404 returned error can't find the container with id 622c53a599c055a022a4dbe2f4b1b1790cfec293c66d2265867c9061e40cf6d4 Jul 15 23:35:55.100648 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.100556 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:35:55.103353 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.103327 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.110690 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.110669 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 23:35:55.111110 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.111094 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 23:35:55.111222 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.111119 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 23:35:55.112536 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.112519 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rljl5\"" Jul 15 23:35:55.112633 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.112556 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 23:35:55.126847 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.126820 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.126998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.126875 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntszp\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.126998 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.126959 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.127108 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.127012 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.127108 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.127055 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.127108 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.127087 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.127231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.127159 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.130796 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.130770 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:35:55.227732 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227699 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.227732 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227747 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227775 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227810 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227845 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227877 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227899 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntszp\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228211 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.227941 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.228983 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.228950 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.230553 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.230528 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.230699 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.230670 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.230832 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.230773 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.230910 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.230864 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1af7a9c9-f783-4f73-9b67-a869f4761843-tls-certs\") pod \"model-serving-api-54d89bd5fc-78f9q\" (UID: \"1af7a9c9-f783-4f73-9b67-a869f4761843\") " pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:55.230910 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.230891 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.236761 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.236730 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.236891 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.236875 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntszp\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp\") pod \"istiod-openshift-gateway-7574459656-btf26\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.414150 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.414046 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:35:55.473466 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.473347 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:35:55.562559 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.562524 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:35:55.631044 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.631012 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-78f9q"] Jul 15 23:35:55.667867 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.667778 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" event={"ID":"1af7a9c9-f783-4f73-9b67-a869f4761843","Type":"ContainerStarted","Data":"0263cbdb973743c11869a774f0d9f112095dbe416dac85949de2382c92728b01"} Jul 15 23:35:55.671049 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.670977 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" event={"ID":"246baebf-fd4f-4007-bcbf-29eddfac4ece","Type":"ContainerStarted","Data":"461e2563da66253d2ba3c61499061f0e8f2fda54c9732053b08c2d375b0a3c7b"} Jul 15 23:35:55.673305 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.673269 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" event={"ID":"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989","Type":"ContainerStarted","Data":"622c53a599c055a022a4dbe2f4b1b1790cfec293c66d2265867c9061e40cf6d4"} Jul 15 23:35:55.675045 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:35:55.675011 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" event={"ID":"671e0999-eb83-489f-908d-875529d5668e","Type":"ContainerStarted","Data":"0928013bbfd52a4a33ba57d383712a6c9280e155a70a2925f7ff516b4aadf6bc"} Jul 15 23:36:00.436636 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.436594 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:36:00.436920 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.436681 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:36:00.705755 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.705629 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" event={"ID":"0c14bfa3-4cfd-444b-a4a0-05c05dcc5989","Type":"ContainerStarted","Data":"6f9515f21b679023518f9515829918da63a4cbde1f523f73bf406eaae87a3fe7"} Jul 15 23:36:00.707633 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.707588 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" event={"ID":"671e0999-eb83-489f-908d-875529d5668e","Type":"ContainerStarted","Data":"f5ab6ccfc8c6066b8f8594b0f266aaa5eb37b6d3438d91ff67e9cffcc097b229"} Jul 15 23:36:00.707874 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.707665 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:36:00.709246 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.709223 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" event={"ID":"1af7a9c9-f783-4f73-9b67-a869f4761843","Type":"ContainerStarted","Data":"539a6b58dd3ce7b574d78ff26b4aa437714470b419a15eff2c60715609d3cc57"} Jul 15 23:36:00.709468 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.709350 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:36:00.711097 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.711077 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" event={"ID":"246baebf-fd4f-4007-bcbf-29eddfac4ece","Type":"ContainerStarted","Data":"42a6bb929751b523e0f69b7efe9583397064f483809913bdca3b65ff0f6c3374"} Jul 15 23:36:00.711209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.711193 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:36:00.722664 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.722609 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bddb6d9b-9bf4t" podStartSLOduration=2.333344347 podStartE2EDuration="7.722587564s" totalImagesPullingTime="5.389243217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:00.72148631 +0000 UTC m=+350.010301732" watchObservedRunningTime="2026-07-15 23:36:00.722587564 +0000 UTC m=+350.011402995" Jul 15 23:36:00.743814 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.743734 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" podStartSLOduration=0.879698424 podStartE2EDuration="5.743712656s" totalImagesPullingTime="4.864014232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:00.741732761 +0000 UTC m=+350.030548190" watchObservedRunningTime="2026-07-15 23:36:00.743712656 +0000 UTC m=+350.032528107" Jul 15 23:36:00.758814 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.758764 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" podStartSLOduration=2.035904966 podStartE2EDuration="6.758748461s" totalImagesPullingTime="4.722843495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:00.75768368 +0000 UTC m=+350.046499121" watchObservedRunningTime="2026-07-15 23:36:00.758748461 +0000 UTC m=+350.047563893" Jul 15 23:36:00.774531 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:00.774470 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" podStartSLOduration=2.361606426 podStartE2EDuration="7.774449025s" totalImagesPullingTime="5.412842599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:00.773200381 +0000 UTC m=+350.062015812" watchObservedRunningTime="2026-07-15 23:36:00.774449025 +0000 UTC m=+350.063264455" Jul 15 23:36:01.717078 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:01.717039 2551 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-btf26 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 23:36:01.717512 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:01.717093 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 23:36:04.716836 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:04.716755 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:36:05.314682 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.314653 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-47wnf"] Jul 15 23:36:05.317630 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.317609 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:05.320581 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.320551 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 23:36:05.320760 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.320643 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-d26hh\"" Jul 15 23:36:05.325945 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.325921 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85"] Jul 15 23:36:05.333255 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.333225 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-47wnf"] Jul 15 23:36:05.333395 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.333354 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.337096 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.336304 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k592q\"" Jul 15 23:36:05.337096 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.336550 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 15 23:36:05.339605 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.339572 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85"] Jul 15 23:36:05.417027 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.416981 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtgv\" (UniqueName: \"kubernetes.io/projected/c8513607-5504-45da-8a33-24b47fc86f6c-kube-api-access-zrtgv\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.417027 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.417030 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdjd\" (UniqueName: \"kubernetes.io/projected/b6d0286f-b670-4834-b456-f1ca01006f82-kube-api-access-zxdjd\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:05.417274 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.417055 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.417274 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.417210 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:05.518741 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.518688 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtgv\" (UniqueName: \"kubernetes.io/projected/c8513607-5504-45da-8a33-24b47fc86f6c-kube-api-access-zrtgv\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.518968 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.518751 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdjd\" (UniqueName: \"kubernetes.io/projected/b6d0286f-b670-4834-b456-f1ca01006f82-kube-api-access-zxdjd\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:05.518968 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.518784 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.518968 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.518852 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:05.518968 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:36:05.518961 2551 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 15 23:36:05.519186 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:36:05.518963 2551 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 15 23:36:05.519186 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:36:05.519044 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert podName:b6d0286f-b670-4834-b456-f1ca01006f82 nodeName:}" failed. No retries permitted until 2026-07-15 23:36:06.019022117 +0000 UTC m=+355.307837530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert") pod "kserve-controller-manager-d6f4b65b8-47wnf" (UID: "b6d0286f-b670-4834-b456-f1ca01006f82") : secret "kserve-webhook-server-cert" not found Jul 15 23:36:05.519186 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:36:05.519077 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert podName:c8513607-5504-45da-8a33-24b47fc86f6c nodeName:}" failed. No retries permitted until 2026-07-15 23:36:06.019057201 +0000 UTC m=+355.307872615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert") pod "llmisvc-controller-manager-777c6bd7d5-49m85" (UID: "c8513607-5504-45da-8a33-24b47fc86f6c") : secret "llmisvc-webhook-server-cert" not found Jul 15 23:36:05.536303 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.536266 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtgv\" (UniqueName: \"kubernetes.io/projected/c8513607-5504-45da-8a33-24b47fc86f6c-kube-api-access-zrtgv\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:05.537644 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:05.537615 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdjd\" (UniqueName: \"kubernetes.io/projected/b6d0286f-b670-4834-b456-f1ca01006f82-kube-api-access-zxdjd\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:06.023729 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.023693 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:06.024149 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.023752 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:06.026300 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.026274 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8513607-5504-45da-8a33-24b47fc86f6c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-49m85\" (UID: \"c8513607-5504-45da-8a33-24b47fc86f6c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:06.026395 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.026269 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6d0286f-b670-4834-b456-f1ca01006f82-cert\") pod \"kserve-controller-manager-d6f4b65b8-47wnf\" (UID: \"b6d0286f-b670-4834-b456-f1ca01006f82\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:06.230545 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.230507 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:06.246456 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.246417 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:06.383573 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.383380 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-47wnf"] Jul 15 23:36:06.385734 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:36:06.385709 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d0286f_b670_4834_b456_f1ca01006f82.slice/crio-aad3cbf8373a1dc68be7fbf6e845c31a2db4fb465500281cac4f46bcc0c22757 WatchSource:0}: Error finding container aad3cbf8373a1dc68be7fbf6e845c31a2db4fb465500281cac4f46bcc0c22757: Status 404 returned error can't find the container with id aad3cbf8373a1dc68be7fbf6e845c31a2db4fb465500281cac4f46bcc0c22757 Jul 15 23:36:06.395733 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.395711 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85"] Jul 15 23:36:06.398411 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:36:06.398385 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8513607_5504_45da_8a33_24b47fc86f6c.slice/crio-af50983d863faf5293e5b642797a9bade8c82b3346408908f467e3fcd854d43a WatchSource:0}: Error finding container af50983d863faf5293e5b642797a9bade8c82b3346408908f467e3fcd854d43a: Status 404 returned error can't find the container with id af50983d863faf5293e5b642797a9bade8c82b3346408908f467e3fcd854d43a Jul 15 23:36:06.736257 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.736161 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" event={"ID":"c8513607-5504-45da-8a33-24b47fc86f6c","Type":"ContainerStarted","Data":"af50983d863faf5293e5b642797a9bade8c82b3346408908f467e3fcd854d43a"} Jul 15 23:36:06.737365 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:06.737346 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" event={"ID":"b6d0286f-b670-4834-b456-f1ca01006f82","Type":"ContainerStarted","Data":"aad3cbf8373a1dc68be7fbf6e845c31a2db4fb465500281cac4f46bcc0c22757"} Jul 15 23:36:08.427189 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:08.427150 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:36:08.427717 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:08.427378 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerName="discovery" containerID="cri-o://42a6bb929751b523e0f69b7efe9583397064f483809913bdca3b65ff0f6c3374" gracePeriod=30 Jul 15 23:36:09.754809 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.754778 2551 generic.go:356] "Generic (PLEG): container finished" podID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerID="42a6bb929751b523e0f69b7efe9583397064f483809913bdca3b65ff0f6c3374" exitCode=0 Jul 15 23:36:09.755189 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.754846 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" event={"ID":"246baebf-fd4f-4007-bcbf-29eddfac4ece","Type":"ContainerDied","Data":"42a6bb929751b523e0f69b7efe9583397064f483809913bdca3b65ff0f6c3374"} Jul 15 23:36:09.755189 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.754901 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" event={"ID":"246baebf-fd4f-4007-bcbf-29eddfac4ece","Type":"ContainerDied","Data":"461e2563da66253d2ba3c61499061f0e8f2fda54c9732053b08c2d375b0a3c7b"} Jul 15 23:36:09.755189 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.754918 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461e2563da66253d2ba3c61499061f0e8f2fda54c9732053b08c2d375b0a3c7b" Jul 15 23:36:09.775555 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.775523 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:36:09.866409 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866371 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866409 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866408 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866439 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntszp\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866457 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866521 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866549 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.866847 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.866681 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap\") pod \"246baebf-fd4f-4007-bcbf-29eddfac4ece\" (UID: \"246baebf-fd4f-4007-bcbf-29eddfac4ece\") " Jul 15 23:36:09.867352 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.867316 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:36:09.868932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.868903 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:36:09.869111 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.868967 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs" (OuterVolumeSpecName: "local-certs") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:36:09.869111 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.869066 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp" (OuterVolumeSpecName: "kube-api-access-ntszp") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "kube-api-access-ntszp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:36:09.869111 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.869080 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token" (OuterVolumeSpecName: "istio-token") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:36:09.869251 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.869110 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts" (OuterVolumeSpecName: "cacerts") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:36:09.869251 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.869144 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "246baebf-fd4f-4007-bcbf-29eddfac4ece" (UID: "246baebf-fd4f-4007-bcbf-29eddfac4ece"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:36:09.967388 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967352 2551 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-kubeconfig\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967388 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967385 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967400 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntszp\" (UniqueName: \"kubernetes.io/projected/246baebf-fd4f-4007-bcbf-29eddfac4ece-kube-api-access-ntszp\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967414 2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-dns-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967429 2551 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/246baebf-fd4f-4007-bcbf-29eddfac4ece-local-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967441 2551 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/246baebf-fd4f-4007-bcbf-29eddfac4ece-cacerts\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:09.967576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:09.967453 2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/246baebf-fd4f-4007-bcbf-29eddfac4ece-istio-csr-ca-configmap\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:10.761167 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.761105 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" event={"ID":"c8513607-5504-45da-8a33-24b47fc86f6c","Type":"ContainerStarted","Data":"6a06a90a7f6626df709edbd9018dc3e0e066d1309a0f274edacd88f58d177110"} Jul 15 23:36:10.761582 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.761193 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:10.762744 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.762720 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" event={"ID":"b6d0286f-b670-4834-b456-f1ca01006f82","Type":"ContainerStarted","Data":"af48eddc5220edab1b6b877361394cf0cac106fc19cbc15daefafb6c83c01bb9"} Jul 15 23:36:10.762847 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.762753 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-btf26" Jul 15 23:36:10.762847 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.762762 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:10.797474 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.797418 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" podStartSLOduration=2.217407711 podStartE2EDuration="5.797398768s" totalImagesPullingTime="3.579991057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:10.796254113 +0000 UTC m=+360.085069546" watchObservedRunningTime="2026-07-15 23:36:10.797398768 +0000 UTC m=+360.086214197" Jul 15 23:36:10.798049 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.798014 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" podStartSLOduration=2.224660064 podStartE2EDuration="5.79800428s" totalImagesPullingTime="3.573344216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:10.781273651 +0000 UTC m=+360.070089080" watchObservedRunningTime="2026-07-15 23:36:10.79800428 +0000 UTC m=+360.086819733" Jul 15 23:36:10.812095 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.812052 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:36:10.815260 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:10.815231 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-btf26"] Jul 15 23:36:11.303879 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:11.303850 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" path="/var/lib/kubelet/pods/246baebf-fd4f-4007-bcbf-29eddfac4ece/volumes" Jul 15 23:36:11.718454 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:11.718363 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-b2mrj" Jul 15 23:36:11.720463 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:11.720441 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-78f9q" Jul 15 23:36:41.771826 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:41.771796 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-49m85" Jul 15 23:36:41.774978 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:41.774954 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-47wnf" Jul 15 23:36:54.546081 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.546038 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:54.546529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.546356 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerName="discovery" Jul 15 23:36:54.546529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.546367 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerName="discovery" Jul 15 23:36:54.546529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.546424 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="246baebf-fd4f-4007-bcbf-29eddfac4ece" containerName="discovery" Jul 15 23:36:54.550841 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.550816 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:54.554875 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.554836 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 23:36:54.555029 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.554842 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9lf7l\"" Jul 15 23:36:54.555355 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.555331 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 23:36:54.558958 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.558931 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:54.631941 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.631902 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hck7f\" (UniqueName: \"kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f\") pod \"kuadrant-operator-catalog-5pthw\" (UID: \"d057af81-4bd7-432c-833f-7247ff48aa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:54.732863 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.732819 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hck7f\" (UniqueName: \"kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f\") pod \"kuadrant-operator-catalog-5pthw\" (UID: \"d057af81-4bd7-432c-833f-7247ff48aa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:54.741084 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.741044 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hck7f\" (UniqueName: \"kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f\") pod \"kuadrant-operator-catalog-5pthw\" (UID: \"d057af81-4bd7-432c-833f-7247ff48aa7a\") " pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:54.863508 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.863406 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:54.902174 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:54.902112 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:55.000324 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.000155 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:55.003228 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:36:55.003195 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd057af81_4bd7_432c_833f_7247ff48aa7a.slice/crio-1a01ed9fdbf4cb2f3c07636539d09c11885f7eb7b6a3d3cd386b03883f06a6e4 WatchSource:0}: Error finding container 1a01ed9fdbf4cb2f3c07636539d09c11885f7eb7b6a3d3cd386b03883f06a6e4: Status 404 returned error can't find the container with id 1a01ed9fdbf4cb2f3c07636539d09c11885f7eb7b6a3d3cd386b03883f06a6e4 Jul 15 23:36:55.112695 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.112660 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-98j7l"] Jul 15 23:36:55.115563 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.115507 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:36:55.123888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.123860 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-98j7l"] Jul 15 23:36:55.136399 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.136361 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp84\" (UniqueName: \"kubernetes.io/projected/1907ec38-9bd2-4434-8c12-09d34406d019-kube-api-access-whp84\") pod \"kuadrant-operator-catalog-98j7l\" (UID: \"1907ec38-9bd2-4434-8c12-09d34406d019\") " pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:36:55.237762 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.237725 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp84\" (UniqueName: \"kubernetes.io/projected/1907ec38-9bd2-4434-8c12-09d34406d019-kube-api-access-whp84\") pod \"kuadrant-operator-catalog-98j7l\" (UID: \"1907ec38-9bd2-4434-8c12-09d34406d019\") " pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:36:55.245497 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.245469 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp84\" (UniqueName: \"kubernetes.io/projected/1907ec38-9bd2-4434-8c12-09d34406d019-kube-api-access-whp84\") pod \"kuadrant-operator-catalog-98j7l\" (UID: \"1907ec38-9bd2-4434-8c12-09d34406d019\") " pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:36:55.426033 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.425936 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:36:55.580410 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.580384 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-98j7l"] Jul 15 23:36:55.582815 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:36:55.582785 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1907ec38_9bd2_4434_8c12_09d34406d019.slice/crio-923d247cb8e71b203f029e8c69fde2943302184fbb1f6e5e169e6ffb03333a64 WatchSource:0}: Error finding container 923d247cb8e71b203f029e8c69fde2943302184fbb1f6e5e169e6ffb03333a64: Status 404 returned error can't find the container with id 923d247cb8e71b203f029e8c69fde2943302184fbb1f6e5e169e6ffb03333a64 Jul 15 23:36:55.965187 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.965145 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" event={"ID":"1907ec38-9bd2-4434-8c12-09d34406d019","Type":"ContainerStarted","Data":"923d247cb8e71b203f029e8c69fde2943302184fbb1f6e5e169e6ffb03333a64"} Jul 15 23:36:55.966687 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:55.966653 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" event={"ID":"d057af81-4bd7-432c-833f-7247ff48aa7a","Type":"ContainerStarted","Data":"1a01ed9fdbf4cb2f3c07636539d09c11885f7eb7b6a3d3cd386b03883f06a6e4"} Jul 15 23:36:57.979001 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:57.978964 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" event={"ID":"1907ec38-9bd2-4434-8c12-09d34406d019","Type":"ContainerStarted","Data":"28dfab3246a50e4268434d5a4244105699c19e2feb928d664bca6db0db379ac7"} Jul 15 23:36:57.980500 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:57.980474 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" event={"ID":"d057af81-4bd7-432c-833f-7247ff48aa7a","Type":"ContainerStarted","Data":"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d"} Jul 15 23:36:57.980640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:57.980558 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" podUID="d057af81-4bd7-432c-833f-7247ff48aa7a" containerName="registry-server" containerID="cri-o://2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d" gracePeriod=2 Jul 15 23:36:57.994812 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:57.994764 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" podStartSLOduration=1.659736614 podStartE2EDuration="2.994747783s" totalImagesPullingTime="1.335011169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:57.994103279 +0000 UTC m=+407.282918709" watchObservedRunningTime="2026-07-15 23:36:57.994747783 +0000 UTC m=+407.283563190" Jul 15 23:36:58.009316 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.009256 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" podStartSLOduration=2.095136794 podStartE2EDuration="4.009235231s" totalImagesPullingTime="1.914098437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:36:58.008277076 +0000 UTC m=+407.297092680" watchObservedRunningTime="2026-07-15 23:36:58.009235231 +0000 UTC m=+407.298050662" Jul 15 23:36:58.226923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.226895 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:58.263407 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.263376 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hck7f\" (UniqueName: \"kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f\") pod \"d057af81-4bd7-432c-833f-7247ff48aa7a\" (UID: \"d057af81-4bd7-432c-833f-7247ff48aa7a\") " Jul 15 23:36:58.265733 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.265698 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f" (OuterVolumeSpecName: "kube-api-access-hck7f") pod "d057af81-4bd7-432c-833f-7247ff48aa7a" (UID: "d057af81-4bd7-432c-833f-7247ff48aa7a"). InnerVolumeSpecName "kube-api-access-hck7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:36:58.364597 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.364551 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hck7f\" (UniqueName: \"kubernetes.io/projected/d057af81-4bd7-432c-833f-7247ff48aa7a-kube-api-access-hck7f\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:36:58.987023 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.986983 2551 generic.go:356] "Generic (PLEG): container finished" podID="d057af81-4bd7-432c-833f-7247ff48aa7a" containerID="2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d" exitCode=0 Jul 15 23:36:58.987023 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.987025 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" Jul 15 23:36:58.987574 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.987069 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" event={"ID":"d057af81-4bd7-432c-833f-7247ff48aa7a","Type":"ContainerDied","Data":"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d"} Jul 15 23:36:58.987574 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.987118 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pthw" event={"ID":"d057af81-4bd7-432c-833f-7247ff48aa7a","Type":"ContainerDied","Data":"1a01ed9fdbf4cb2f3c07636539d09c11885f7eb7b6a3d3cd386b03883f06a6e4"} Jul 15 23:36:58.987574 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.987167 2551 scope.go:117] "RemoveContainer" containerID="2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d" Jul 15 23:36:58.996768 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.996749 2551 scope.go:117] "RemoveContainer" containerID="2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d" Jul 15 23:36:58.997106 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:36:58.997082 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d\": container with ID starting with 2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d not found: ID does not exist" containerID="2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d" Jul 15 23:36:58.997239 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:58.997116 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d"} err="failed to get container status \"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d\": rpc error: code = NotFound desc = could not find container \"2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d\": container with ID starting with 2f3ff144dff1476b389aedf84527cf535ecbf6cfea8f424a9629e7233b8fef8d not found: ID does not exist" Jul 15 23:36:59.009075 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:59.009037 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:59.012942 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:59.012910 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pthw"] Jul 15 23:36:59.302380 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:36:59.302350 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d057af81-4bd7-432c-833f-7247ff48aa7a" path="/var/lib/kubelet/pods/d057af81-4bd7-432c-833f-7247ff48aa7a/volumes" Jul 15 23:37:05.426904 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:05.426862 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:37:05.427375 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:05.426928 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:37:05.450307 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:05.450279 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:37:06.041346 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:06.041316 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-98j7l" Jul 15 23:37:09.941243 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.941207 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p"] Jul 15 23:37:09.941638 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.941482 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d057af81-4bd7-432c-833f-7247ff48aa7a" containerName="registry-server" Jul 15 23:37:09.941638 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.941492 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d057af81-4bd7-432c-833f-7247ff48aa7a" containerName="registry-server" Jul 15 23:37:09.941638 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.941570 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d057af81-4bd7-432c-833f-7247ff48aa7a" containerName="registry-server" Jul 15 23:37:09.943592 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.943570 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:09.946353 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.946327 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-grzcx\"" Jul 15 23:37:09.952387 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:09.952359 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p"] Jul 15 23:37:10.057937 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.057897 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.058099 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.057958 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.058099 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.058043 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tl6\" (UniqueName: \"kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.159420 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.159385 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.159568 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.159444 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.159568 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.159473 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tl6\" (UniqueName: \"kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.159826 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.159805 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.159861 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.159844 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.168051 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.168021 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tl6\" (UniqueName: \"kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.254009 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.253913 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:10.395102 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.395074 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p"] Jul 15 23:37:10.397565 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:10.397534 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5622e5_706f_4405_b8bd_b5045c7fe0a4.slice/crio-62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d WatchSource:0}: Error finding container 62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d: Status 404 returned error can't find the container with id 62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d Jul 15 23:37:10.541746 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.541710 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql"] Jul 15 23:37:10.544097 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.544076 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.553903 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.553870 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql"] Jul 15 23:37:10.664519 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.664485 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.664697 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.664533 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf95x\" (UniqueName: \"kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.664697 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.664561 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.765213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.765163 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.765213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.765211 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf95x\" (UniqueName: \"kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.765478 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.765229 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.765627 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.765605 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.765672 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.765609 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.773952 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.773923 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf95x\" (UniqueName: \"kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.871178 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.871053 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:10.947149 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.945587 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg"] Jul 15 23:37:10.956480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.956443 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:10.958002 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:10.957958 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg"] Jul 15 23:37:11.032071 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.032049 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql"] Jul 15 23:37:11.034297 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:11.034264 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7334b077_8fd1_4485_bd23_b5b00990e4b0.slice/crio-4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8 WatchSource:0}: Error finding container 4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8: Status 404 returned error can't find the container with id 4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8 Jul 15 23:37:11.040340 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.040301 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" event={"ID":"7334b077-8fd1-4485-bd23-b5b00990e4b0","Type":"ContainerStarted","Data":"4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8"} Jul 15 23:37:11.041866 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.041836 2551 generic.go:356] "Generic (PLEG): container finished" podID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerID="2d8ba9dcc15cd2c5cd8cad7160b3c67d22001bca65d54c5771d6010be2e98a53" exitCode=0 Jul 15 23:37:11.041962 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.041897 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" event={"ID":"0d5622e5-706f-4405-b8bd-b5045c7fe0a4","Type":"ContainerDied","Data":"2d8ba9dcc15cd2c5cd8cad7160b3c67d22001bca65d54c5771d6010be2e98a53"} Jul 15 23:37:11.041962 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.041940 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" event={"ID":"0d5622e5-706f-4405-b8bd-b5045c7fe0a4","Type":"ContainerStarted","Data":"62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d"} Jul 15 23:37:11.067269 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.067232 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.067448 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.067283 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mqt\" (UniqueName: \"kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.067448 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.067370 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.168049 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.168009 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.168256 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.168182 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.168256 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.168223 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mqt\" (UniqueName: \"kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.168472 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.168442 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.168540 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.168506 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.176799 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.176769 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mqt\" (UniqueName: \"kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.268685 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.268636 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:11.345675 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.345203 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5"] Jul 15 23:37:11.348353 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.348325 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.362041 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.361848 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5"] Jul 15 23:37:11.419576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.419549 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg"] Jul 15 23:37:11.421939 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:11.421907 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d761059_19c1_4239_b837_b73e3ecb0e7c.slice/crio-f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952 WatchSource:0}: Error finding container f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952: Status 404 returned error can't find the container with id f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952 Jul 15 23:37:11.470872 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.470817 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.471072 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.470918 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rmz\" (UniqueName: \"kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.471072 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.470985 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.572193 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.572156 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.572373 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.572248 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rmz\" (UniqueName: \"kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.572373 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.572306 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.572493 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.572431 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.572627 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.572608 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.581174 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.581142 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rmz\" (UniqueName: \"kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.667964 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.667928 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:11.825246 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:11.825221 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5"] Jul 15 23:37:11.836111 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:11.836081 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2931f9_bfc0_44a5_9c67_37642969172b.slice/crio-9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d WatchSource:0}: Error finding container 9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d: Status 404 returned error can't find the container with id 9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d Jul 15 23:37:12.047556 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.047522 2551 generic.go:356] "Generic (PLEG): container finished" podID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerID="182f46fd5ba2d24a5b2d5b0fce66f6ed587fd0b86c6992b4c77c5f0e03701bc4" exitCode=0 Jul 15 23:37:12.047969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.047607 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" event={"ID":"4d761059-19c1-4239-b837-b73e3ecb0e7c","Type":"ContainerDied","Data":"182f46fd5ba2d24a5b2d5b0fce66f6ed587fd0b86c6992b4c77c5f0e03701bc4"} Jul 15 23:37:12.047969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.047664 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" event={"ID":"4d761059-19c1-4239-b837-b73e3ecb0e7c","Type":"ContainerStarted","Data":"f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952"} Jul 15 23:37:12.049365 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.049341 2551 generic.go:356] "Generic (PLEG): container finished" podID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerID="e80f2a46bda2ad9b5e4581de099e2b4c5fa54d93d919aeb33eee619cb9f145a0" exitCode=0 Jul 15 23:37:12.049475 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.049408 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" event={"ID":"7334b077-8fd1-4485-bd23-b5b00990e4b0","Type":"ContainerDied","Data":"e80f2a46bda2ad9b5e4581de099e2b4c5fa54d93d919aeb33eee619cb9f145a0"} Jul 15 23:37:12.051506 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.051483 2551 generic.go:356] "Generic (PLEG): container finished" podID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerID="8d50681c1da9075d948227885663422f259844875221df3f2d332403019a2a66" exitCode=0 Jul 15 23:37:12.051601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.051513 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" event={"ID":"0d5622e5-706f-4405-b8bd-b5045c7fe0a4","Type":"ContainerDied","Data":"8d50681c1da9075d948227885663422f259844875221df3f2d332403019a2a66"} Jul 15 23:37:12.053299 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.053273 2551 generic.go:356] "Generic (PLEG): container finished" podID="da2931f9-bfc0-44a5-9c67-37642969172b" containerID="abe6d7c551563196f1a0029b6cab646301b19646b3bc1d4d48d77acc59d6c313" exitCode=0 Jul 15 23:37:12.053383 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.053351 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" event={"ID":"da2931f9-bfc0-44a5-9c67-37642969172b","Type":"ContainerDied","Data":"abe6d7c551563196f1a0029b6cab646301b19646b3bc1d4d48d77acc59d6c313"} Jul 15 23:37:12.053442 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:12.053400 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" event={"ID":"da2931f9-bfc0-44a5-9c67-37642969172b","Type":"ContainerStarted","Data":"9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d"} Jul 15 23:37:13.060601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.060509 2551 generic.go:356] "Generic (PLEG): container finished" podID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerID="48096b41d7899e065d9a3d8c0d3cfa777ad6871dc082c93eb322d9a521d7a9f8" exitCode=0 Jul 15 23:37:13.061061 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.060611 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" event={"ID":"7334b077-8fd1-4485-bd23-b5b00990e4b0","Type":"ContainerDied","Data":"48096b41d7899e065d9a3d8c0d3cfa777ad6871dc082c93eb322d9a521d7a9f8"} Jul 15 23:37:13.062932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.062908 2551 generic.go:356] "Generic (PLEG): container finished" podID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerID="d456580c68300e44d94ce5e4befc6f9f81f4ee94c0238071d6f3532a5122d996" exitCode=0 Jul 15 23:37:13.063034 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.062984 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" event={"ID":"0d5622e5-706f-4405-b8bd-b5045c7fe0a4","Type":"ContainerDied","Data":"d456580c68300e44d94ce5e4befc6f9f81f4ee94c0238071d6f3532a5122d996"} Jul 15 23:37:13.064811 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.064793 2551 generic.go:356] "Generic (PLEG): container finished" podID="da2931f9-bfc0-44a5-9c67-37642969172b" containerID="20605467fd8c9d2dd32ce8fbfbc68142a01682177f7e722df4f55b145f022d77" exitCode=0 Jul 15 23:37:13.064892 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.064874 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" event={"ID":"da2931f9-bfc0-44a5-9c67-37642969172b","Type":"ContainerDied","Data":"20605467fd8c9d2dd32ce8fbfbc68142a01682177f7e722df4f55b145f022d77"} Jul 15 23:37:13.066981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.066957 2551 generic.go:356] "Generic (PLEG): container finished" podID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerID="cd140c217494041aeb77c36875eab32f76a7e8f010fdf32440030c75c72b2c1b" exitCode=0 Jul 15 23:37:13.067050 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:13.067014 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" event={"ID":"4d761059-19c1-4239-b837-b73e3ecb0e7c","Type":"ContainerDied","Data":"cd140c217494041aeb77c36875eab32f76a7e8f010fdf32440030c75c72b2c1b"} Jul 15 23:37:14.074184 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.074145 2551 generic.go:356] "Generic (PLEG): container finished" podID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerID="6b6dfc993b7b7638c1e8ebbf60fc89eae030aa413c6aec824e813b641e9c988f" exitCode=0 Jul 15 23:37:14.074589 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.074228 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" event={"ID":"4d761059-19c1-4239-b837-b73e3ecb0e7c","Type":"ContainerDied","Data":"6b6dfc993b7b7638c1e8ebbf60fc89eae030aa413c6aec824e813b641e9c988f"} Jul 15 23:37:14.076336 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.076302 2551 generic.go:356] "Generic (PLEG): container finished" podID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerID="8c0c98550fc905c59e160a619dc3b842dd815b168d0374717b8220897450abeb" exitCode=0 Jul 15 23:37:14.076494 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.076345 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" event={"ID":"7334b077-8fd1-4485-bd23-b5b00990e4b0","Type":"ContainerDied","Data":"8c0c98550fc905c59e160a619dc3b842dd815b168d0374717b8220897450abeb"} Jul 15 23:37:14.078561 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.078532 2551 generic.go:356] "Generic (PLEG): container finished" podID="da2931f9-bfc0-44a5-9c67-37642969172b" containerID="46630dd8202175fb3a3976bdbaca93af026cea5b425eeacb22b66e5935a8687a" exitCode=0 Jul 15 23:37:14.078681 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.078607 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" event={"ID":"da2931f9-bfc0-44a5-9c67-37642969172b","Type":"ContainerDied","Data":"46630dd8202175fb3a3976bdbaca93af026cea5b425eeacb22b66e5935a8687a"} Jul 15 23:37:14.212264 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.212235 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:14.295964 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.295923 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle\") pod \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " Jul 15 23:37:14.296197 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.295985 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4tl6\" (UniqueName: \"kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6\") pod \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " Jul 15 23:37:14.296197 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.296014 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util\") pod \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\" (UID: \"0d5622e5-706f-4405-b8bd-b5045c7fe0a4\") " Jul 15 23:37:14.296490 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.296460 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle" (OuterVolumeSpecName: "bundle") pod "0d5622e5-706f-4405-b8bd-b5045c7fe0a4" (UID: "0d5622e5-706f-4405-b8bd-b5045c7fe0a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:14.298150 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.298107 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6" (OuterVolumeSpecName: "kube-api-access-p4tl6") pod "0d5622e5-706f-4405-b8bd-b5045c7fe0a4" (UID: "0d5622e5-706f-4405-b8bd-b5045c7fe0a4"). InnerVolumeSpecName "kube-api-access-p4tl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:14.301464 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.301415 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util" (OuterVolumeSpecName: "util") pod "0d5622e5-706f-4405-b8bd-b5045c7fe0a4" (UID: "0d5622e5-706f-4405-b8bd-b5045c7fe0a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:14.397017 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.396915 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tl6\" (UniqueName: \"kubernetes.io/projected/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-kube-api-access-p4tl6\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:14.397017 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.396949 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:14.397017 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:14.396965 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d5622e5-706f-4405-b8bd-b5045c7fe0a4-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.085971 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.085945 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" Jul 15 23:37:15.086404 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.085992 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p" event={"ID":"0d5622e5-706f-4405-b8bd-b5045c7fe0a4","Type":"ContainerDied","Data":"62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d"} Jul 15 23:37:15.086404 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.086036 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e60b9f1e61bf548284bfe779c9a6c89fa6b836b0840b8b3b7c6566f9e8da3d" Jul 15 23:37:15.238509 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.238479 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:15.274690 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.274659 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:15.279402 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.279377 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:15.305450 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.305423 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7rmz\" (UniqueName: \"kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz\") pod \"da2931f9-bfc0-44a5-9c67-37642969172b\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " Jul 15 23:37:15.305583 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.305468 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle\") pod \"da2931f9-bfc0-44a5-9c67-37642969172b\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " Jul 15 23:37:15.305850 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.305608 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util\") pod \"da2931f9-bfc0-44a5-9c67-37642969172b\" (UID: \"da2931f9-bfc0-44a5-9c67-37642969172b\") " Jul 15 23:37:15.307074 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.307016 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle" (OuterVolumeSpecName: "bundle") pod "da2931f9-bfc0-44a5-9c67-37642969172b" (UID: "da2931f9-bfc0-44a5-9c67-37642969172b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.309551 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.309511 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz" (OuterVolumeSpecName: "kube-api-access-s7rmz") pod "da2931f9-bfc0-44a5-9c67-37642969172b" (UID: "da2931f9-bfc0-44a5-9c67-37642969172b"). InnerVolumeSpecName "kube-api-access-s7rmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:15.313467 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.313431 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util" (OuterVolumeSpecName: "util") pod "da2931f9-bfc0-44a5-9c67-37642969172b" (UID: "da2931f9-bfc0-44a5-9c67-37642969172b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.406906 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.406800 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf95x\" (UniqueName: \"kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x\") pod \"7334b077-8fd1-4485-bd23-b5b00990e4b0\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " Jul 15 23:37:15.406906 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.406878 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util\") pod \"7334b077-8fd1-4485-bd23-b5b00990e4b0\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " Jul 15 23:37:15.407172 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.406910 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle\") pod \"4d761059-19c1-4239-b837-b73e3ecb0e7c\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " Jul 15 23:37:15.407172 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.406936 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86mqt\" (UniqueName: \"kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt\") pod \"4d761059-19c1-4239-b837-b73e3ecb0e7c\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " Jul 15 23:37:15.407172 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.406967 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util\") pod \"4d761059-19c1-4239-b837-b73e3ecb0e7c\" (UID: \"4d761059-19c1-4239-b837-b73e3ecb0e7c\") " Jul 15 23:37:15.407172 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407059 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle\") pod \"7334b077-8fd1-4485-bd23-b5b00990e4b0\" (UID: \"7334b077-8fd1-4485-bd23-b5b00990e4b0\") " Jul 15 23:37:15.407364 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407298 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.407364 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407319 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rmz\" (UniqueName: \"kubernetes.io/projected/da2931f9-bfc0-44a5-9c67-37642969172b-kube-api-access-s7rmz\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.407364 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407332 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da2931f9-bfc0-44a5-9c67-37642969172b-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.407627 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407553 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle" (OuterVolumeSpecName: "bundle") pod "7334b077-8fd1-4485-bd23-b5b00990e4b0" (UID: "7334b077-8fd1-4485-bd23-b5b00990e4b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.407691 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.407642 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle" (OuterVolumeSpecName: "bundle") pod "4d761059-19c1-4239-b837-b73e3ecb0e7c" (UID: "4d761059-19c1-4239-b837-b73e3ecb0e7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.409171 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.409094 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x" (OuterVolumeSpecName: "kube-api-access-zf95x") pod "7334b077-8fd1-4485-bd23-b5b00990e4b0" (UID: "7334b077-8fd1-4485-bd23-b5b00990e4b0"). InnerVolumeSpecName "kube-api-access-zf95x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:15.409435 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.409410 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt" (OuterVolumeSpecName: "kube-api-access-86mqt") pod "4d761059-19c1-4239-b837-b73e3ecb0e7c" (UID: "4d761059-19c1-4239-b837-b73e3ecb0e7c"). InnerVolumeSpecName "kube-api-access-86mqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:15.413486 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.413454 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util" (OuterVolumeSpecName: "util") pod "7334b077-8fd1-4485-bd23-b5b00990e4b0" (UID: "7334b077-8fd1-4485-bd23-b5b00990e4b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.413561 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.413455 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util" (OuterVolumeSpecName: "util") pod "4d761059-19c1-4239-b837-b73e3ecb0e7c" (UID: "4d761059-19c1-4239-b837-b73e3ecb0e7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:15.507931 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507888 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.507931 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507925 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.507931 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507938 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mqt\" (UniqueName: \"kubernetes.io/projected/4d761059-19c1-4239-b837-b73e3ecb0e7c-kube-api-access-86mqt\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.508192 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507952 2551 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d761059-19c1-4239-b837-b73e3ecb0e7c-util\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.508192 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507961 2551 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7334b077-8fd1-4485-bd23-b5b00990e4b0-bundle\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:15.508192 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:15.507969 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf95x\" (UniqueName: \"kubernetes.io/projected/7334b077-8fd1-4485-bd23-b5b00990e4b0-kube-api-access-zf95x\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:16.093060 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.093027 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" Jul 15 23:37:16.093480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.093026 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql" event={"ID":"7334b077-8fd1-4485-bd23-b5b00990e4b0","Type":"ContainerDied","Data":"4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8"} Jul 15 23:37:16.093480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.093175 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4acf4300e113c754c2d0fa9936fcf7f0b471dd1c1b6d20c444ec395295bd9df8" Jul 15 23:37:16.095117 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.095095 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" Jul 15 23:37:16.095256 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.095138 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5" event={"ID":"da2931f9-bfc0-44a5-9c67-37642969172b","Type":"ContainerDied","Data":"9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d"} Jul 15 23:37:16.095256 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.095194 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9618b82bb5d4251f1ee0ed4153568fe05763090fc2f316dc1857e44362c06d6d" Jul 15 23:37:16.097207 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.097180 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" event={"ID":"4d761059-19c1-4239-b837-b73e3ecb0e7c","Type":"ContainerDied","Data":"f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952"} Jul 15 23:37:16.097341 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.097234 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f609e9387554d91d5906ad8f43ae3afc070f534ce40d685692ef0dca23af0952" Jul 15 23:37:16.097341 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:16.097239 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg" Jul 15 23:37:21.670576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670535 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bpds8"] Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670823 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670834 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670853 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670859 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670865 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670872 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670885 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670890 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670897 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670902 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670908 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670913 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670919 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670924 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670929 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670934 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670940 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670946 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670950 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670955 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670961 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670965 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="util" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670973 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.670978 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="pull" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.671038 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7334b077-8fd1-4485-bd23-b5b00990e4b0" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.671047 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d761059-19c1-4239-b837-b73e3ecb0e7c" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.671053 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2931f9-bfc0-44a5-9c67-37642969172b" containerName="extract" Jul 15 23:37:21.671053 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.671060 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5622e5-706f-4405-b8bd-b5045c7fe0a4" containerName="extract" Jul 15 23:37:21.675563 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.675534 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:21.678475 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.678449 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gpthr\"" Jul 15 23:37:21.691024 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.690991 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bpds8"] Jul 15 23:37:21.761369 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.761315 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqq2r\" (UniqueName: \"kubernetes.io/projected/d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd-kube-api-access-hqq2r\") pod \"authorino-operator-75cc8d5cb7-bpds8\" (UID: \"d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:21.862478 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.862437 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqq2r\" (UniqueName: \"kubernetes.io/projected/d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd-kube-api-access-hqq2r\") pod \"authorino-operator-75cc8d5cb7-bpds8\" (UID: \"d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:21.871247 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.871217 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqq2r\" (UniqueName: \"kubernetes.io/projected/d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd-kube-api-access-hqq2r\") pod \"authorino-operator-75cc8d5cb7-bpds8\" (UID: \"d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:21.986212 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:21.986087 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:22.123892 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:22.123863 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bpds8"] Jul 15 23:37:22.125908 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:22.125878 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd437fb2a_a7ba_4f17_a5e6_dd2e2b8e16fd.slice/crio-51ffbac4321b4f1f29438fa02b683fc7328941e1b8b15fd76a27dc9cd18f0459 WatchSource:0}: Error finding container 51ffbac4321b4f1f29438fa02b683fc7328941e1b8b15fd76a27dc9cd18f0459: Status 404 returned error can't find the container with id 51ffbac4321b4f1f29438fa02b683fc7328941e1b8b15fd76a27dc9cd18f0459 Jul 15 23:37:23.134781 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:23.134748 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" event={"ID":"d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd","Type":"ContainerStarted","Data":"51ffbac4321b4f1f29438fa02b683fc7328941e1b8b15fd76a27dc9cd18f0459"} Jul 15 23:37:25.146875 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.146827 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" event={"ID":"d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd","Type":"ContainerStarted","Data":"5d1de9c1064e4e1d96b02a57ae75a3457abb1231f1e167e0fed0bb4e3ddb3739"} Jul 15 23:37:25.147288 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.146966 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:25.644509 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.644453 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" podStartSLOduration=2.297343085 podStartE2EDuration="4.644432968s" totalImagesPullingTime="2.347089883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:25.162891682 +0000 UTC m=+434.451707110" watchObservedRunningTime="2026-07-15 23:37:25.644432968 +0000 UTC m=+434.933248397" Jul 15 23:37:25.645474 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.645456 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc"] Jul 15 23:37:25.648721 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.648703 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:25.651386 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.651359 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qkttx\"" Jul 15 23:37:25.657440 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.657410 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc"] Jul 15 23:37:25.796620 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.796578 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn48\" (UniqueName: \"kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48\") pod \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" (UID: \"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:25.897462 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.897365 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn48\" (UniqueName: \"kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48\") pod \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" (UID: \"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:25.908565 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.908529 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn48\" (UniqueName: \"kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48\") pod \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" (UID: \"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:25.960182 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:25.960115 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:26.100091 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:26.100062 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc"] Jul 15 23:37:26.102832 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:26.102793 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd439f5b3_2e0f_4fe9_b6fd_57da16ded3f8.slice/crio-994ffdbb33bac278ebffebfc49f4fc5cff83637b7b417c51b265dcca42a49b6f WatchSource:0}: Error finding container 994ffdbb33bac278ebffebfc49f4fc5cff83637b7b417c51b265dcca42a49b6f: Status 404 returned error can't find the container with id 994ffdbb33bac278ebffebfc49f4fc5cff83637b7b417c51b265dcca42a49b6f Jul 15 23:37:26.153997 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:26.153902 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" event={"ID":"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8","Type":"ContainerStarted","Data":"994ffdbb33bac278ebffebfc49f4fc5cff83637b7b417c51b265dcca42a49b6f"} Jul 15 23:37:28.166554 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:28.166454 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" event={"ID":"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8","Type":"ContainerStarted","Data":"f4b0664385fd68c19c9deca5c1d7f4972a4f9858d26215eea739a20374276958"} Jul 15 23:37:28.166943 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:28.166571 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:28.186926 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:28.186872 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" podStartSLOduration=1.382400112 podStartE2EDuration="3.186854752s" totalImagesPullingTime="1.80445464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:28.1868441 +0000 UTC m=+437.475659511" watchObservedRunningTime="2026-07-15 23:37:28.186854752 +0000 UTC m=+437.475670177" Jul 15 23:37:30.395374 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.395327 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l"] Jul 15 23:37:30.398929 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.398904 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.401909 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.401887 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-44h9l\"" Jul 15 23:37:30.410099 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.410073 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l"] Jul 15 23:37:30.533319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.533278 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdznv\" (UniqueName: \"kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.533319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.533321 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.634493 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.634449 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdznv\" (UniqueName: \"kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.634493 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.634495 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.634876 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.634860 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.645811 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.645730 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdznv\" (UniqueName: \"kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.709528 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.709492 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:30.862906 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:30.862867 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l"] Jul 15 23:37:30.863688 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:30.863663 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecdbb99_5585_4c12_819a_324fa7278de9.slice/crio-2235cb6756fa7f63452c414601f91bab304babfbf4601af48f36e9b5169e2858 WatchSource:0}: Error finding container 2235cb6756fa7f63452c414601f91bab304babfbf4601af48f36e9b5169e2858: Status 404 returned error can't find the container with id 2235cb6756fa7f63452c414601f91bab304babfbf4601af48f36e9b5169e2858 Jul 15 23:37:31.181028 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:31.180993 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" event={"ID":"2ecdbb99-5585-4c12-819a-324fa7278de9","Type":"ContainerStarted","Data":"2235cb6756fa7f63452c414601f91bab304babfbf4601af48f36e9b5169e2858"} Jul 15 23:37:36.156876 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:36.156844 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bpds8" Jul 15 23:37:36.208507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:36.208473 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" event={"ID":"2ecdbb99-5585-4c12-819a-324fa7278de9","Type":"ContainerStarted","Data":"af57588e75d8d34502687bf1a1a9ca3fcb8122de8bce2d2b05b3b15d8828ff2d"} Jul 15 23:37:36.208660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:36.208573 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:36.228281 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:36.228227 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" podStartSLOduration=1.060561144 podStartE2EDuration="6.228211293s" totalImagesPullingTime="5.167650149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:36.227034322 +0000 UTC m=+445.515849751" watchObservedRunningTime="2026-07-15 23:37:36.228211293 +0000 UTC m=+445.517026723" Jul 15 23:37:39.173256 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:39.173223 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:47.216571 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:47.216539 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.030054 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.030008 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l"] Jul 15 23:37:49.030571 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.030264 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" containerName="manager" containerID="cri-o://af57588e75d8d34502687bf1a1a9ca3fcb8122de8bce2d2b05b3b15d8828ff2d" gracePeriod=2 Jul 15 23:37:49.043058 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.043025 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l"] Jul 15 23:37:49.055695 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.055654 2551 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2ecdbb99-5585-4c12-819a-324fa7278de9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.059740 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.059690 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:37:49.060217 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.060194 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" containerName="manager" Jul 15 23:37:49.060394 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.060382 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" containerName="manager" Jul 15 23:37:49.060586 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.060563 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" containerName="manager" Jul 15 23:37:49.063081 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.063024 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc"] Jul 15 23:37:49.063313 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.063160 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.063440 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.063368 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerName="manager" containerID="cri-o://f4b0664385fd68c19c9deca5c1d7f4972a4f9858d26215eea739a20374276958" gracePeriod=2 Jul 15 23:37:49.065692 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.065639 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.066117 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.066091 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc"] Jul 15 23:37:49.067937 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.067711 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:49.079291 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.079253 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:37:49.086034 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.086003 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.086106 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.086049 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrl8\" (UniqueName: \"kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.091119 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.091084 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7"] Jul 15 23:37:49.091488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.091471 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerName="manager" Jul 15 23:37:49.091573 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.091492 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerName="manager" Jul 15 23:37:49.091621 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.091579 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerName="manager" Jul 15 23:37:49.093499 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.093476 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:49.096564 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.096527 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.098761 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.098717 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:49.101164 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.101098 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jrm9l\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.107323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.107287 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7"] Jul 15 23:37:49.127026 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:37:49.126992 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ftqxc\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:49.187305 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.187266 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vj8b\" (UniqueName: \"kubernetes.io/projected/05a7ad7a-92c3-4778-9ed2-cf28db3df85a-kube-api-access-5vj8b\") pod \"limitador-operator-controller-manager-859d7b78bd-dtsw7\" (UID: \"05a7ad7a-92c3-4778-9ed2-cf28db3df85a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:49.187475 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.187324 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.187475 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.187375 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrl8\" (UniqueName: \"kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.187747 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.187727 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.196703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.196666 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrl8\" (UniqueName: \"kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xpzh9\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.273529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.273496 2551 generic.go:356] "Generic (PLEG): container finished" podID="2ecdbb99-5585-4c12-819a-324fa7278de9" containerID="af57588e75d8d34502687bf1a1a9ca3fcb8122de8bce2d2b05b3b15d8828ff2d" exitCode=0 Jul 15 23:37:49.275341 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.275313 2551 generic.go:356] "Generic (PLEG): container finished" podID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerID="f4b0664385fd68c19c9deca5c1d7f4972a4f9858d26215eea739a20374276958" exitCode=0 Jul 15 23:37:49.287766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.287743 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vj8b\" (UniqueName: \"kubernetes.io/projected/05a7ad7a-92c3-4778-9ed2-cf28db3df85a-kube-api-access-5vj8b\") pod \"limitador-operator-controller-manager-859d7b78bd-dtsw7\" (UID: \"05a7ad7a-92c3-4778-9ed2-cf28db3df85a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:49.297513 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.297486 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vj8b\" (UniqueName: \"kubernetes.io/projected/05a7ad7a-92c3-4778-9ed2-cf28db3df85a-kube-api-access-5vj8b\") pod \"limitador-operator-controller-manager-859d7b78bd-dtsw7\" (UID: \"05a7ad7a-92c3-4778-9ed2-cf28db3df85a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:49.309198 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.309169 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:49.312084 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.312059 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:49.441508 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.441457 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:49.449928 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.449885 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:49.489199 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.489167 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdznv\" (UniqueName: \"kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv\") pod \"2ecdbb99-5585-4c12-819a-324fa7278de9\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " Jul 15 23:37:49.489369 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.489218 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hn48\" (UniqueName: \"kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48\") pod \"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8\" (UID: \"d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8\") " Jul 15 23:37:49.489369 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.489244 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume\") pod \"2ecdbb99-5585-4c12-819a-324fa7278de9\" (UID: \"2ecdbb99-5585-4c12-819a-324fa7278de9\") " Jul 15 23:37:49.489798 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.489765 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2ecdbb99-5585-4c12-819a-324fa7278de9" (UID: "2ecdbb99-5585-4c12-819a-324fa7278de9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:37:49.492042 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.491997 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48" (OuterVolumeSpecName: "kube-api-access-4hn48") pod "d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" (UID: "d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8"). InnerVolumeSpecName "kube-api-access-4hn48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:49.492209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.492108 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv" (OuterVolumeSpecName: "kube-api-access-cdznv") pod "2ecdbb99-5585-4c12-819a-324fa7278de9" (UID: "2ecdbb99-5585-4c12-819a-324fa7278de9"). InnerVolumeSpecName "kube-api-access-cdznv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:37:49.589708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.589675 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hn48\" (UniqueName: \"kubernetes.io/projected/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8-kube-api-access-4hn48\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:49.589708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.589704 2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ecdbb99-5585-4c12-819a-324fa7278de9-extensions-socket-volume\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:49.589708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.589714 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdznv\" (UniqueName: \"kubernetes.io/projected/2ecdbb99-5585-4c12-819a-324fa7278de9-kube-api-access-cdznv\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:37:49.601492 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.601455 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:37:49.603267 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:49.603238 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb1e275_d3a7_4d30_8bf8_ec71c576f1fd.slice/crio-c5b01b77e61bfba94e1bd8c1fdc9b008565f44a129f1b0f5009b6ad7fea1bb17 WatchSource:0}: Error finding container c5b01b77e61bfba94e1bd8c1fdc9b008565f44a129f1b0f5009b6ad7fea1bb17: Status 404 returned error can't find the container with id c5b01b77e61bfba94e1bd8c1fdc9b008565f44a129f1b0f5009b6ad7fea1bb17 Jul 15 23:37:49.621522 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:49.621496 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7"] Jul 15 23:37:49.623880 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:37:49.623853 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a7ad7a_92c3_4778_9ed2_cf28db3df85a.slice/crio-d98e42886a27d5e7b02eda12ce8dc55779f6af2c27872258a3b21831dfdd4648 WatchSource:0}: Error finding container d98e42886a27d5e7b02eda12ce8dc55779f6af2c27872258a3b21831dfdd4648: Status 404 returned error can't find the container with id d98e42886a27d5e7b02eda12ce8dc55779f6af2c27872258a3b21831dfdd4648 Jul 15 23:37:50.171730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.171672 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.30:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 23:37:50.281522 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.281484 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" event={"ID":"05a7ad7a-92c3-4778-9ed2-cf28db3df85a","Type":"ContainerStarted","Data":"88af55b693c651240ce68ca92af7ab2b04ad8ed12d3c08a5c6df3823313edb0b"} Jul 15 23:37:50.281730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.281548 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" event={"ID":"05a7ad7a-92c3-4778-9ed2-cf28db3df85a","Type":"ContainerStarted","Data":"d98e42886a27d5e7b02eda12ce8dc55779f6af2c27872258a3b21831dfdd4648"} Jul 15 23:37:50.281730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.281668 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:37:50.283349 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.283319 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" event={"ID":"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd","Type":"ContainerStarted","Data":"d5df2e1a9ac0d4f84b638d135c7354ab00d302a9e27794dc2506a31bbf85b77e"} Jul 15 23:37:50.283514 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.283365 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" event={"ID":"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd","Type":"ContainerStarted","Data":"c5b01b77e61bfba94e1bd8c1fdc9b008565f44a129f1b0f5009b6ad7fea1bb17"} Jul 15 23:37:50.283514 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.283397 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:37:50.284776 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.284755 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ftqxc" Jul 15 23:37:50.284898 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.284791 2551 scope.go:117] "RemoveContainer" containerID="f4b0664385fd68c19c9deca5c1d7f4972a4f9858d26215eea739a20374276958" Jul 15 23:37:50.286432 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.286413 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jrm9l" Jul 15 23:37:50.294476 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.294451 2551 scope.go:117] "RemoveContainer" containerID="af57588e75d8d34502687bf1a1a9ca3fcb8122de8bce2d2b05b3b15d8828ff2d" Jul 15 23:37:50.302775 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.302676 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" podStartSLOduration=1.30265632 podStartE2EDuration="1.30265632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:50.301246023 +0000 UTC m=+459.590061453" watchObservedRunningTime="2026-07-15 23:37:50.30265632 +0000 UTC m=+459.591471746" Jul 15 23:37:50.322529 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:50.322471 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" podStartSLOduration=1.3224513469999999 podStartE2EDuration="1.322451347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:37:50.320709818 +0000 UTC m=+459.609525248" watchObservedRunningTime="2026-07-15 23:37:50.322451347 +0000 UTC m=+459.611266776" Jul 15 23:37:51.303097 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:51.303062 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ecdbb99-5585-4c12-819a-324fa7278de9" path="/var/lib/kubelet/pods/2ecdbb99-5585-4c12-819a-324fa7278de9/volumes" Jul 15 23:37:51.303477 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:37:51.303404 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8" path="/var/lib/kubelet/pods/d439f5b3-2e0f-4fe9-b6fd-57da16ded3f8/volumes" Jul 15 23:38:01.295987 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:01.295952 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dtsw7" Jul 15 23:38:01.296569 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:01.296059 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:38:18.187241 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.187202 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:38:18.187701 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.187400 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" podUID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" containerName="manager" containerID="cri-o://d5df2e1a9ac0d4f84b638d135c7354ab00d302a9e27794dc2506a31bbf85b77e" gracePeriod=10 Jul 15 23:38:18.447820 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.447790 2551 generic.go:356] "Generic (PLEG): container finished" podID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" containerID="d5df2e1a9ac0d4f84b638d135c7354ab00d302a9e27794dc2506a31bbf85b77e" exitCode=0 Jul 15 23:38:18.447973 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.447867 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" event={"ID":"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd","Type":"ContainerDied","Data":"d5df2e1a9ac0d4f84b638d135c7354ab00d302a9e27794dc2506a31bbf85b77e"} Jul 15 23:38:18.462928 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.462902 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:38:18.507600 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.507566 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghrl8\" (UniqueName: \"kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8\") pod \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " Jul 15 23:38:18.507792 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.507618 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume\") pod \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\" (UID: \"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd\") " Jul 15 23:38:18.508157 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.508102 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" (UID: "9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:38:18.509850 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.509813 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8" (OuterVolumeSpecName: "kube-api-access-ghrl8") pod "9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" (UID: "9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd"). InnerVolumeSpecName "kube-api-access-ghrl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:18.535115 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.535079 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:38:18.535410 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.535397 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" containerName="manager" Jul 15 23:38:18.535467 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.535412 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" containerName="manager" Jul 15 23:38:18.535467 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.535463 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" containerName="manager" Jul 15 23:38:18.537347 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.537330 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.556743 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.555656 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:38:18.609035 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.608994 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.609231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.609048 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfg8\" (UniqueName: \"kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.609231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.609120 2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-extensions-socket-volume\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:38:18.609231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.609155 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrl8\" (UniqueName: \"kubernetes.io/projected/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd-kube-api-access-ghrl8\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:38:18.710435 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.710341 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.710435 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.710395 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfg8\" (UniqueName: \"kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.710769 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.710747 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.727018 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.726976 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfg8\" (UniqueName: \"kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8\") pod \"kuadrant-operator-controller-manager-86878c769f-d492q\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:18.847476 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:18.847431 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:19.009785 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.009757 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:38:19.012441 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:38:19.012414 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2001ffa4_8784_46ff_9fa6_b5731cd921ad.slice/crio-efda183907d3865c37124e96587c0816a8ccff25d0bee856868358daae8679ea WatchSource:0}: Error finding container efda183907d3865c37124e96587c0816a8ccff25d0bee856868358daae8679ea: Status 404 returned error can't find the container with id efda183907d3865c37124e96587c0816a8ccff25d0bee856868358daae8679ea Jul 15 23:38:19.453875 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.453843 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" Jul 15 23:38:19.453875 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.453863 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9" event={"ID":"9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd","Type":"ContainerDied","Data":"c5b01b77e61bfba94e1bd8c1fdc9b008565f44a129f1b0f5009b6ad7fea1bb17"} Jul 15 23:38:19.454354 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.453926 2551 scope.go:117] "RemoveContainer" containerID="d5df2e1a9ac0d4f84b638d135c7354ab00d302a9e27794dc2506a31bbf85b77e" Jul 15 23:38:19.458733 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.458705 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" event={"ID":"2001ffa4-8784-46ff-9fa6-b5731cd921ad","Type":"ContainerStarted","Data":"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3"} Jul 15 23:38:19.458866 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.458769 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:19.458866 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.458796 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" event={"ID":"2001ffa4-8784-46ff-9fa6-b5731cd921ad","Type":"ContainerStarted","Data":"efda183907d3865c37124e96587c0816a8ccff25d0bee856868358daae8679ea"} Jul 15 23:38:19.502897 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.502837 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" podStartSLOduration=1.502818769 podStartE2EDuration="1.502818769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:19.502587107 +0000 UTC m=+488.791402536" watchObservedRunningTime="2026-07-15 23:38:19.502818769 +0000 UTC m=+488.791634198" Jul 15 23:38:19.538662 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.538629 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:38:19.547231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:19.547183 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xpzh9"] Jul 15 23:38:21.302429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:21.302394 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd" path="/var/lib/kubelet/pods/9fb1e275-d3a7-4d30-8bf8-ec71c576f1fd/volumes" Jul 15 23:38:30.469014 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:30.468981 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:38:37.505961 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.505920 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:38:37.509521 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.509213 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.512396 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.512368 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-grzcx\"" Jul 15 23:38:37.512396 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.512388 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 23:38:37.520794 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.520765 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:38:37.547997 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.547958 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:38:37.562336 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.562297 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a67338b-2730-49d4-b97d-2b5d444696d1-config-file\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.562567 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.562473 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhwv\" (UniqueName: \"kubernetes.io/projected/8a67338b-2730-49d4-b97d-2b5d444696d1-kube-api-access-shhwv\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.663898 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.663866 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhwv\" (UniqueName: \"kubernetes.io/projected/8a67338b-2730-49d4-b97d-2b5d444696d1-kube-api-access-shhwv\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.664079 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.663910 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a67338b-2730-49d4-b97d-2b5d444696d1-config-file\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.664721 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.664698 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8a67338b-2730-49d4-b97d-2b5d444696d1-config-file\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.672267 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.672236 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhwv\" (UniqueName: \"kubernetes.io/projected/8a67338b-2730-49d4-b97d-2b5d444696d1-kube-api-access-shhwv\") pod \"limitador-limitador-54d6d988d5-ql6l2\" (UID: \"8a67338b-2730-49d4-b97d-2b5d444696d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.821477 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.821389 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:37.959528 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:37.959497 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:38:37.962631 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:38:37.962605 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a67338b_2730_49d4_b97d_2b5d444696d1.slice/crio-50cd2df28411b85e908c9a611742505171a1d9c343743b510a29758358e30261 WatchSource:0}: Error finding container 50cd2df28411b85e908c9a611742505171a1d9c343743b510a29758358e30261: Status 404 returned error can't find the container with id 50cd2df28411b85e908c9a611742505171a1d9c343743b510a29758358e30261 Jul 15 23:38:38.200993 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.200911 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:38.205175 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.205153 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:38.209343 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.209292 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-66thq\"" Jul 15 23:38:38.212056 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.212028 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:38.270460 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.270426 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b4kh\" (UniqueName: \"kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh\") pod \"authorino-58cd955675-ssmq7\" (UID: \"aa00da80-45a4-42e0-a0f8-df0883df1d41\") " pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:38.370891 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.370852 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b4kh\" (UniqueName: \"kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh\") pod \"authorino-58cd955675-ssmq7\" (UID: \"aa00da80-45a4-42e0-a0f8-df0883df1d41\") " pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:38.381388 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.381353 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b4kh\" (UniqueName: \"kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh\") pod \"authorino-58cd955675-ssmq7\" (UID: \"aa00da80-45a4-42e0-a0f8-df0883df1d41\") " pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:38.398515 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.398474 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:38:38.402077 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.402048 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:38:38.407645 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.407616 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:38:38.471997 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.471912 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rs6\" (UniqueName: \"kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6\") pod \"authorino-75d556bd77-j9n9x\" (UID: \"f601bbec-a6b6-416b-9fa1-8e1aa9de4426\") " pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:38:38.517029 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.516990 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:38.561060 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.561022 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" event={"ID":"8a67338b-2730-49d4-b97d-2b5d444696d1","Type":"ContainerStarted","Data":"50cd2df28411b85e908c9a611742505171a1d9c343743b510a29758358e30261"} Jul 15 23:38:38.572761 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.572720 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rs6\" (UniqueName: \"kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6\") pod \"authorino-75d556bd77-j9n9x\" (UID: \"f601bbec-a6b6-416b-9fa1-8e1aa9de4426\") " pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:38:38.583403 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.583370 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rs6\" (UniqueName: \"kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6\") pod \"authorino-75d556bd77-j9n9x\" (UID: \"f601bbec-a6b6-416b-9fa1-8e1aa9de4426\") " pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:38:38.707524 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.707175 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:38.713018 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.712997 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:38:38.905557 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:38.905529 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:38:38.910618 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:38:38.910583 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf601bbec_a6b6_416b_9fa1_8e1aa9de4426.slice/crio-b664176a5c259dbc08e9e6d733d632f6d03dd060148ae93687f776cdfd6fd739 WatchSource:0}: Error finding container b664176a5c259dbc08e9e6d733d632f6d03dd060148ae93687f776cdfd6fd739: Status 404 returned error can't find the container with id b664176a5c259dbc08e9e6d733d632f6d03dd060148ae93687f776cdfd6fd739 Jul 15 23:38:39.570186 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:39.570118 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-j9n9x" event={"ID":"f601bbec-a6b6-416b-9fa1-8e1aa9de4426","Type":"ContainerStarted","Data":"b664176a5c259dbc08e9e6d733d632f6d03dd060148ae93687f776cdfd6fd739"} Jul 15 23:38:39.573376 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:39.573332 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ssmq7" event={"ID":"aa00da80-45a4-42e0-a0f8-df0883df1d41","Type":"ContainerStarted","Data":"8ff0d7e9c69c527fbda4828505718387f8ca236546033ee2d73c0446cc88b024"} Jul 15 23:38:43.598842 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.598804 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-j9n9x" event={"ID":"f601bbec-a6b6-416b-9fa1-8e1aa9de4426","Type":"ContainerStarted","Data":"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79"} Jul 15 23:38:43.600366 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.600339 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" event={"ID":"8a67338b-2730-49d4-b97d-2b5d444696d1","Type":"ContainerStarted","Data":"0bc64e0bd27e85d9af1b68c5a48fc6240b0e476cc176950a9f49c8a383419457"} Jul 15 23:38:43.600503 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.600395 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:38:43.601745 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.601720 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ssmq7" event={"ID":"aa00da80-45a4-42e0-a0f8-df0883df1d41","Type":"ContainerStarted","Data":"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0"} Jul 15 23:38:43.615229 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.615181 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-j9n9x" podStartSLOduration=1.667531028 podStartE2EDuration="5.615163945s" totalImagesPullingTime="3.947632917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:43.613213474 +0000 UTC m=+512.902028902" watchObservedRunningTime="2026-07-15 23:38:43.615163945 +0000 UTC m=+512.903979371" Jul 15 23:38:43.627995 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.627946 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-ssmq7" podStartSLOduration=1.486379933 podStartE2EDuration="5.627926137s" totalImagesPullingTime="4.141546204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:43.627334891 +0000 UTC m=+512.916150321" watchObservedRunningTime="2026-07-15 23:38:43.627926137 +0000 UTC m=+512.916741568" Jul 15 23:38:43.646601 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.646551 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" podStartSLOduration=1.756613293 podStartE2EDuration="6.646532551s" totalImagesPullingTime="4.889919258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:38:43.645808098 +0000 UTC m=+512.934623529" watchObservedRunningTime="2026-07-15 23:38:43.646532551 +0000 UTC m=+512.935347981" Jul 15 23:38:43.660567 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:43.660532 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:45.611899 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:45.611851 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-ssmq7" podUID="aa00da80-45a4-42e0-a0f8-df0883df1d41" containerName="authorino" containerID="cri-o://0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0" gracePeriod=30 Jul 15 23:38:45.868616 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:45.868542 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:45.942538 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:45.942498 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7b4kh\" (UniqueName: \"kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh\") pod \"aa00da80-45a4-42e0-a0f8-df0883df1d41\" (UID: \"aa00da80-45a4-42e0-a0f8-df0883df1d41\") " Jul 15 23:38:45.944538 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:45.944512 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh" (OuterVolumeSpecName: "kube-api-access-7b4kh") pod "aa00da80-45a4-42e0-a0f8-df0883df1d41" (UID: "aa00da80-45a4-42e0-a0f8-df0883df1d41"). InnerVolumeSpecName "kube-api-access-7b4kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:38:46.044046 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.044015 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b4kh\" (UniqueName: \"kubernetes.io/projected/aa00da80-45a4-42e0-a0f8-df0883df1d41-kube-api-access-7b4kh\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:38:46.617555 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.617516 2551 generic.go:356] "Generic (PLEG): container finished" podID="aa00da80-45a4-42e0-a0f8-df0883df1d41" containerID="0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0" exitCode=0 Jul 15 23:38:46.617555 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.617558 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ssmq7" Jul 15 23:38:46.618112 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.617592 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ssmq7" event={"ID":"aa00da80-45a4-42e0-a0f8-df0883df1d41","Type":"ContainerDied","Data":"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0"} Jul 15 23:38:46.618112 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.617625 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ssmq7" event={"ID":"aa00da80-45a4-42e0-a0f8-df0883df1d41","Type":"ContainerDied","Data":"8ff0d7e9c69c527fbda4828505718387f8ca236546033ee2d73c0446cc88b024"} Jul 15 23:38:46.618112 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.617646 2551 scope.go:117] "RemoveContainer" containerID="0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0" Jul 15 23:38:46.626910 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.626888 2551 scope.go:117] "RemoveContainer" containerID="0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0" Jul 15 23:38:46.627234 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:38:46.627205 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0\": container with ID starting with 0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0 not found: ID does not exist" containerID="0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0" Jul 15 23:38:46.627307 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.627245 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0"} err="failed to get container status \"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0\": rpc error: code = NotFound desc = could not find container \"0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0\": container with ID starting with 0ad6ffb61231d0d7a33a2f785d0e769f5e0e3a31de7f3697c9945e0f7063a9d0 not found: ID does not exist" Jul 15 23:38:46.642429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.642398 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:46.648505 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:46.648471 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-ssmq7"] Jul 15 23:38:47.302641 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:47.302607 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa00da80-45a4-42e0-a0f8-df0883df1d41" path="/var/lib/kubelet/pods/aa00da80-45a4-42e0-a0f8-df0883df1d41/volumes" Jul 15 23:38:54.608056 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:38:54.608024 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ql6l2" Jul 15 23:39:10.933323 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.933231 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:10.933862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.933586 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa00da80-45a4-42e0-a0f8-df0883df1d41" containerName="authorino" Jul 15 23:39:10.933862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.933598 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa00da80-45a4-42e0-a0f8-df0883df1d41" containerName="authorino" Jul 15 23:39:10.933862 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.933669 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa00da80-45a4-42e0-a0f8-df0883df1d41" containerName="authorino" Jul 15 23:39:10.940832 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.940802 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:10.942895 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:10.942867 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:11.061514 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.061474 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7q8\" (UniqueName: \"kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8\") pod \"authorino-5d4b67c446-7fgk4\" (UID: \"3ca41696-7c8e-4685-94bf-6f16a64bacd1\") " pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:11.162853 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.162816 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7q8\" (UniqueName: \"kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8\") pod \"authorino-5d4b67c446-7fgk4\" (UID: \"3ca41696-7c8e-4685-94bf-6f16a64bacd1\") " pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:11.172905 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.172868 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7q8\" (UniqueName: \"kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8\") pod \"authorino-5d4b67c446-7fgk4\" (UID: \"3ca41696-7c8e-4685-94bf-6f16a64bacd1\") " pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:11.184168 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.184062 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:11.184342 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.184328 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:11.209023 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.208988 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:39:11.211983 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.211960 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.214547 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.214523 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 23:39:11.219968 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.219944 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:39:11.263575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.263541 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chm77\" (UniqueName: \"kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.263734 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.263592 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.327322 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.327296 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:11.330155 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:39:11.330086 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca41696_7c8e_4685_94bf_6f16a64bacd1.slice/crio-2e12e58ba371222bb1abea3bcee53d882fb9392cc0ef44ef4ff36a8f9d854dde WatchSource:0}: Error finding container 2e12e58ba371222bb1abea3bcee53d882fb9392cc0ef44ef4ff36a8f9d854dde: Status 404 returned error can't find the container with id 2e12e58ba371222bb1abea3bcee53d882fb9392cc0ef44ef4ff36a8f9d854dde Jul 15 23:39:11.364793 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.364755 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.365038 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.365013 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chm77\" (UniqueName: \"kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.367248 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.367226 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.372543 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.372516 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chm77\" (UniqueName: \"kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77\") pod \"authorino-fb5b544f5-c5cvn\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.523187 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.523118 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:39:11.656184 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.656110 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:39:11.658346 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:39:11.658308 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89bcc97_cd48_4ccc_baf5_1ee26b688327.slice/crio-541d741028e650cb66ad21d3061c382066a82c263604213567b45df70cadca05 WatchSource:0}: Error finding container 541d741028e650cb66ad21d3061c382066a82c263604213567b45df70cadca05: Status 404 returned error can't find the container with id 541d741028e650cb66ad21d3061c382066a82c263604213567b45df70cadca05 Jul 15 23:39:11.742012 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.741980 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" event={"ID":"3ca41696-7c8e-4685-94bf-6f16a64bacd1","Type":"ContainerStarted","Data":"2e12e58ba371222bb1abea3bcee53d882fb9392cc0ef44ef4ff36a8f9d854dde"} Jul 15 23:39:11.743015 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:11.742993 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" event={"ID":"b89bcc97-cd48-4ccc-baf5-1ee26b688327","Type":"ContainerStarted","Data":"541d741028e650cb66ad21d3061c382066a82c263604213567b45df70cadca05"} Jul 15 23:39:12.749551 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.749514 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" event={"ID":"3ca41696-7c8e-4685-94bf-6f16a64bacd1","Type":"ContainerStarted","Data":"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758"} Jul 15 23:39:12.749953 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.749572 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" podUID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" containerName="authorino" containerID="cri-o://15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758" gracePeriod=30 Jul 15 23:39:12.751156 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.751112 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" event={"ID":"b89bcc97-cd48-4ccc-baf5-1ee26b688327","Type":"ContainerStarted","Data":"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74"} Jul 15 23:39:12.763740 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.763692 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" podStartSLOduration=2.401305097 podStartE2EDuration="2.763674124s" totalImagesPullingTime="362.369027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:39:12.762750566 +0000 UTC m=+542.051565994" watchObservedRunningTime="2026-07-15 23:39:12.763674124 +0000 UTC m=+542.052489550" Jul 15 23:39:12.776615 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.776554 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" podStartSLOduration=1.377519571 podStartE2EDuration="1.776537144s" totalImagesPullingTime="399.017573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:39:12.776165308 +0000 UTC m=+542.064980734" watchObservedRunningTime="2026-07-15 23:39:12.776537144 +0000 UTC m=+542.065352572" Jul 15 23:39:12.801390 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.801348 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:39:12.801637 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:12.801601 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-j9n9x" podUID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" containerName="authorino" containerID="cri-o://ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79" gracePeriod=30 Jul 15 23:39:13.057705 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.057677 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:13.060794 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.060772 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:39:13.179988 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.179960 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx7q8\" (UniqueName: \"kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8\") pod \"3ca41696-7c8e-4685-94bf-6f16a64bacd1\" (UID: \"3ca41696-7c8e-4685-94bf-6f16a64bacd1\") " Jul 15 23:39:13.180112 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.180054 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2rs6\" (UniqueName: \"kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6\") pod \"f601bbec-a6b6-416b-9fa1-8e1aa9de4426\" (UID: \"f601bbec-a6b6-416b-9fa1-8e1aa9de4426\") " Jul 15 23:39:13.182208 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.182163 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6" (OuterVolumeSpecName: "kube-api-access-v2rs6") pod "f601bbec-a6b6-416b-9fa1-8e1aa9de4426" (UID: "f601bbec-a6b6-416b-9fa1-8e1aa9de4426"). InnerVolumeSpecName "kube-api-access-v2rs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:39:13.182303 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.182219 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8" (OuterVolumeSpecName: "kube-api-access-wx7q8") pod "3ca41696-7c8e-4685-94bf-6f16a64bacd1" (UID: "3ca41696-7c8e-4685-94bf-6f16a64bacd1"). InnerVolumeSpecName "kube-api-access-wx7q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:39:13.281150 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.281089 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx7q8\" (UniqueName: \"kubernetes.io/projected/3ca41696-7c8e-4685-94bf-6f16a64bacd1-kube-api-access-wx7q8\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:39:13.281331 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.281120 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rs6\" (UniqueName: \"kubernetes.io/projected/f601bbec-a6b6-416b-9fa1-8e1aa9de4426-kube-api-access-v2rs6\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:39:13.757710 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.757680 2551 generic.go:356] "Generic (PLEG): container finished" podID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" containerID="15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758" exitCode=0 Jul 15 23:39:13.758212 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.757762 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" Jul 15 23:39:13.758212 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.757771 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" event={"ID":"3ca41696-7c8e-4685-94bf-6f16a64bacd1","Type":"ContainerDied","Data":"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758"} Jul 15 23:39:13.758212 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.757818 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7fgk4" event={"ID":"3ca41696-7c8e-4685-94bf-6f16a64bacd1","Type":"ContainerDied","Data":"2e12e58ba371222bb1abea3bcee53d882fb9392cc0ef44ef4ff36a8f9d854dde"} Jul 15 23:39:13.758212 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.757835 2551 scope.go:117] "RemoveContainer" containerID="15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758" Jul 15 23:39:13.759161 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.758984 2551 generic.go:356] "Generic (PLEG): container finished" podID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" containerID="ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79" exitCode=0 Jul 15 23:39:13.759161 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.759050 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-j9n9x" Jul 15 23:39:13.759161 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.759073 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-j9n9x" event={"ID":"f601bbec-a6b6-416b-9fa1-8e1aa9de4426","Type":"ContainerDied","Data":"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79"} Jul 15 23:39:13.759161 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.759113 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-j9n9x" event={"ID":"f601bbec-a6b6-416b-9fa1-8e1aa9de4426","Type":"ContainerDied","Data":"b664176a5c259dbc08e9e6d733d632f6d03dd060148ae93687f776cdfd6fd739"} Jul 15 23:39:13.766587 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.766569 2551 scope.go:117] "RemoveContainer" containerID="15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758" Jul 15 23:39:13.766888 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:39:13.766861 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758\": container with ID starting with 15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758 not found: ID does not exist" containerID="15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758" Jul 15 23:39:13.766958 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.766901 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758"} err="failed to get container status \"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758\": rpc error: code = NotFound desc = could not find container \"15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758\": container with ID starting with 15768336206f447e6a90bfee6219a88abe88ba9994d51829e5666c1b09332758 not found: ID does not exist" Jul 15 23:39:13.766958 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.766925 2551 scope.go:117] "RemoveContainer" containerID="ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79" Jul 15 23:39:13.774144 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.774094 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:13.775733 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.775709 2551 scope.go:117] "RemoveContainer" containerID="ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79" Jul 15 23:39:13.776099 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:39:13.776074 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79\": container with ID starting with ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79 not found: ID does not exist" containerID="ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79" Jul 15 23:39:13.776276 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.776109 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79"} err="failed to get container status \"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79\": rpc error: code = NotFound desc = could not find container \"ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79\": container with ID starting with ab670ad52f17b825f2b4acd2c6681e7ff8ad03396444b4faca0668ce0e3b7e79 not found: ID does not exist" Jul 15 23:39:13.778002 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.777981 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7fgk4"] Jul 15 23:39:13.790542 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.790503 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:39:13.794094 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:13.794067 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-j9n9x"] Jul 15 23:39:15.303143 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:15.303104 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" path="/var/lib/kubelet/pods/3ca41696-7c8e-4685-94bf-6f16a64bacd1/volumes" Jul 15 23:39:15.303504 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:15.303444 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" path="/var/lib/kubelet/pods/f601bbec-a6b6-416b-9fa1-8e1aa9de4426/volumes" Jul 15 23:39:52.774347 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:39:52.774296 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:40:38.134431 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:40:38.134340 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:41:11.513503 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:11.513450 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:41:18.617517 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:18.617463 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:41:43.015925 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.015885 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn"] Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016247 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" containerName="authorino" Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016257 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" containerName="authorino" Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016270 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" containerName="authorino" Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016276 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" containerName="authorino" Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016328 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f601bbec-a6b6-416b-9fa1-8e1aa9de4426" containerName="authorino" Jul 15 23:41:43.016488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.016339 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca41696-7c8e-4685-94bf-6f16a64bacd1" containerName="authorino" Jul 15 23:41:43.019523 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.019501 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.023607 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.023582 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t4j8f\"" Jul 15 23:41:43.023607 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.023582 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 23:41:43.023843 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.023829 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 23:41:43.023901 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.023860 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 23:41:43.029789 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.029767 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn"] Jul 15 23:41:43.146003 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.145963 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.146003 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.146008 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc96\" (UniqueName: \"kubernetes.io/projected/3887d43c-ce7a-4f93-a05c-78e853c2af88-kube-api-access-jcc96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.146298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.146028 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3887d43c-ce7a-4f93-a05c-78e853c2af88-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.146298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.146096 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.146298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.146183 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.146298 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.146241 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247393 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247347 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247602 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247424 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247602 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247455 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc96\" (UniqueName: \"kubernetes.io/projected/3887d43c-ce7a-4f93-a05c-78e853c2af88-kube-api-access-jcc96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247602 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247480 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3887d43c-ce7a-4f93-a05c-78e853c2af88-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247602 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247517 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247812 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247635 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247870 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247837 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247870 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247858 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.247965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.247911 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.249712 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.249693 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3887d43c-ce7a-4f93-a05c-78e853c2af88-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.250025 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.250009 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3887d43c-ce7a-4f93-a05c-78e853c2af88-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.255754 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.255722 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc96\" (UniqueName: \"kubernetes.io/projected/3887d43c-ce7a-4f93-a05c-78e853c2af88-kube-api-access-jcc96\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn\" (UID: \"3887d43c-ce7a-4f93-a05c-78e853c2af88\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.330393 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.330305 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:43.470933 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.470895 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn"] Jul 15 23:41:43.472875 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:41:43.472838 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3887d43c_ce7a_4f93_a05c_78e853c2af88.slice/crio-0cd5313a37dc2a80b1069ae0de727a1b21ef59fc8f80d6005d906502b4feb724 WatchSource:0}: Error finding container 0cd5313a37dc2a80b1069ae0de727a1b21ef59fc8f80d6005d906502b4feb724: Status 404 returned error can't find the container with id 0cd5313a37dc2a80b1069ae0de727a1b21ef59fc8f80d6005d906502b4feb724 Jul 15 23:41:43.474422 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.474398 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:41:43.514632 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:43.514599 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" event={"ID":"3887d43c-ce7a-4f93-a05c-78e853c2af88","Type":"ContainerStarted","Data":"0cd5313a37dc2a80b1069ae0de727a1b21ef59fc8f80d6005d906502b4feb724"} Jul 15 23:41:44.811308 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:44.811267 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:41:45.525634 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:45.525593 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" event={"ID":"3887d43c-ce7a-4f93-a05c-78e853c2af88","Type":"ContainerStarted","Data":"551f9022253771853e1fb59f503f4283f907d80ed536f6402a6f0acd1a585755"} Jul 15 23:41:45.545022 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:45.544970 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" podStartSLOduration=2.539903747 podStartE2EDuration="3.544953703s" totalImagesPullingTime="1.005049956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:41:45.543737611 +0000 UTC m=+694.832553041" watchObservedRunningTime="2026-07-15 23:41:45.544953703 +0000 UTC m=+694.833769131" Jul 15 23:41:47.317969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.317929 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw"] Jul 15 23:41:47.321544 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.321523 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.324173 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.324150 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 23:41:47.330604 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.330580 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw"] Jul 15 23:41:47.486999 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.486965 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.487213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.487024 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f1faa0-9677-46d6-8b56-c93e873be608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.487213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.487070 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.487213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.487088 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.487213 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.487110 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lc4\" (UniqueName: \"kubernetes.io/projected/d7f1faa0-9677-46d6-8b56-c93e873be608-kube-api-access-z5lc4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.487358 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.487229 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.587887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587789 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.587887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587850 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f1faa0-9677-46d6-8b56-c93e873be608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.587887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587873 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.587887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587889 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.588254 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587912 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lc4\" (UniqueName: \"kubernetes.io/projected/d7f1faa0-9677-46d6-8b56-c93e873be608-kube-api-access-z5lc4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.588254 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.587943 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.588403 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.588307 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.588403 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.588318 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.588403 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.588376 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.590261 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.590235 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f1faa0-9677-46d6-8b56-c93e873be608-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.590503 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.590486 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f1faa0-9677-46d6-8b56-c93e873be608-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.596201 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.596176 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lc4\" (UniqueName: \"kubernetes.io/projected/d7f1faa0-9677-46d6-8b56-c93e873be608-kube-api-access-z5lc4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cclqw\" (UID: \"d7f1faa0-9677-46d6-8b56-c93e873be608\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.632282 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.632227 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:47.776655 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:47.776632 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw"] Jul 15 23:41:47.778724 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:41:47.778694 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f1faa0_9677_46d6_8b56_c93e873be608.slice/crio-5c2bc3bb65dd2b8f118e6201199ea64eaf5bda50e73e7dfd23e95259a8935d0a WatchSource:0}: Error finding container 5c2bc3bb65dd2b8f118e6201199ea64eaf5bda50e73e7dfd23e95259a8935d0a: Status 404 returned error can't find the container with id 5c2bc3bb65dd2b8f118e6201199ea64eaf5bda50e73e7dfd23e95259a8935d0a Jul 15 23:41:48.542824 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:48.542787 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" event={"ID":"d7f1faa0-9677-46d6-8b56-c93e873be608","Type":"ContainerStarted","Data":"fa8707ddfb7b95de8e33a78ce778e9d713049959071dd57c94cfe7fefb876aeb"} Jul 15 23:41:48.543252 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:48.542850 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" event={"ID":"d7f1faa0-9677-46d6-8b56-c93e873be608","Type":"ContainerStarted","Data":"5c2bc3bb65dd2b8f118e6201199ea64eaf5bda50e73e7dfd23e95259a8935d0a"} Jul 15 23:41:48.561180 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:48.561106 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" podStartSLOduration=1.407857167 podStartE2EDuration="1.561087248s" totalImagesPullingTime="153.230081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:41:48.560059817 +0000 UTC m=+697.848875246" watchObservedRunningTime="2026-07-15 23:41:48.561087248 +0000 UTC m=+697.849902677" Jul 15 23:41:53.331305 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:53.331264 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:53.331717 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:53.331334 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:53.343665 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:53.343634 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:53.581555 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:53.581450 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn" Jul 15 23:41:55.823227 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:55.823183 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:41:57.620953 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.620893 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s"] Jul 15 23:41:57.624547 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.624527 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.627377 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.627353 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 23:41:57.632399 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.632375 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:57.632543 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.632454 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:57.635010 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.634981 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s"] Jul 15 23:41:57.645419 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.645392 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:57.659338 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659298 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.659507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659349 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.659507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659389 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.659507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659418 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.659622 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659514 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb534609-c1bb-430a-830c-5f9030d66cc4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.659622 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.659558 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnp2h\" (UniqueName: \"kubernetes.io/projected/fb534609-c1bb-430a-830c-5f9030d66cc4-kube-api-access-dnp2h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760659 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760625 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb534609-c1bb-430a-830c-5f9030d66cc4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760659 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760669 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnp2h\" (UniqueName: \"kubernetes.io/projected/fb534609-c1bb-430a-830c-5f9030d66cc4-kube-api-access-dnp2h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760695 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760731 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760751 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.760932 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.760791 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.761255 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.761229 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.761354 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.761258 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.761354 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.761319 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.763961 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.763931 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb534609-c1bb-430a-830c-5f9030d66cc4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.764479 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.764454 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb534609-c1bb-430a-830c-5f9030d66cc4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.769673 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.769642 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnp2h\" (UniqueName: \"kubernetes.io/projected/fb534609-c1bb-430a-830c-5f9030d66cc4-kube-api-access-dnp2h\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s\" (UID: \"fb534609-c1bb-430a-830c-5f9030d66cc4\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:57.935694 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:57.935599 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:41:58.082086 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:58.082043 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s"] Jul 15 23:41:58.597243 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:58.597155 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" event={"ID":"fb534609-c1bb-430a-830c-5f9030d66cc4","Type":"ContainerStarted","Data":"4aea6f63872d93af344b9719828d645bd10d37f05f7e85608ab670c91a14e74f"} Jul 15 23:41:58.597243 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:58.597201 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" event={"ID":"fb534609-c1bb-430a-830c-5f9030d66cc4","Type":"ContainerStarted","Data":"fa31879c14f2f508c9d0c57330b3c0b1a462de00c6b952e0456e977ae1175c07"} Jul 15 23:41:58.608937 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:58.608908 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cclqw" Jul 15 23:41:58.615778 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:41:58.615731 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" podStartSLOduration=1.436550837 podStartE2EDuration="1.61571303s" totalImagesPullingTime="179.162193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:41:58.613818313 +0000 UTC m=+707.902633743" watchObservedRunningTime="2026-07-15 23:41:58.61571303 +0000 UTC m=+707.904528487" Jul 15 23:42:01.631319 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:01.631275 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:42:07.936776 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:07.936684 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:42:07.936776 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:07.936745 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:42:07.949315 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:07.949284 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:42:08.660308 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:08.660272 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s" Jul 15 23:42:11.157038 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:42:11.157012 2551 scope.go:117] "RemoveContainer" containerID="42a6bb929751b523e0f69b7efe9583397064f483809913bdca3b65ff0f6c3374" Jul 15 23:43:24.474445 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.474406 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b9cbfc9bb-wkpxp"] Jul 15 23:43:24.477209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.477187 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.483991 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.483958 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9cbfc9bb-wkpxp"] Jul 15 23:43:24.619590 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.619552 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrp82\" (UniqueName: \"kubernetes.io/projected/2f6e3c18-44a7-462a-8516-4108314df640-kube-api-access-lrp82\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.619590 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.619595 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f6e3c18-44a7-462a-8516-4108314df640-tls-cert\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.721101 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.721053 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrp82\" (UniqueName: \"kubernetes.io/projected/2f6e3c18-44a7-462a-8516-4108314df640-kube-api-access-lrp82\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.721101 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.721104 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f6e3c18-44a7-462a-8516-4108314df640-tls-cert\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.723707 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.723680 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2f6e3c18-44a7-462a-8516-4108314df640-tls-cert\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.729229 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.729155 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrp82\" (UniqueName: \"kubernetes.io/projected/2f6e3c18-44a7-462a-8516-4108314df640-kube-api-access-lrp82\") pod \"authorino-7b9cbfc9bb-wkpxp\" (UID: \"2f6e3c18-44a7-462a-8516-4108314df640\") " pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.788158 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.788091 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" Jul 15 23:43:24.934062 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:24.934025 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9cbfc9bb-wkpxp"] Jul 15 23:43:24.937615 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:43:24.937586 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f6e3c18_44a7_462a_8516_4108314df640.slice/crio-e406d4381d294fb6c4520c9afd3ac50d99f69c0cb9425d8deaa884d24156b99a WatchSource:0}: Error finding container e406d4381d294fb6c4520c9afd3ac50d99f69c0cb9425d8deaa884d24156b99a: Status 404 returned error can't find the container with id e406d4381d294fb6c4520c9afd3ac50d99f69c0cb9425d8deaa884d24156b99a Jul 15 23:43:25.033069 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:25.033030 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" event={"ID":"2f6e3c18-44a7-462a-8516-4108314df640","Type":"ContainerStarted","Data":"e406d4381d294fb6c4520c9afd3ac50d99f69c0cb9425d8deaa884d24156b99a"} Jul 15 23:43:26.042720 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.042653 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" event={"ID":"2f6e3c18-44a7-462a-8516-4108314df640","Type":"ContainerStarted","Data":"f51740a5fc0de8df4260825c656fee1f8d4bc51638ea9556c67980b256f4a207"} Jul 15 23:43:26.063156 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.063043 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b9cbfc9bb-wkpxp" podStartSLOduration=1.794146732 podStartE2EDuration="2.063002696s" totalImagesPullingTime="268.855964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:43:26.061831258 +0000 UTC m=+795.350646686" watchObservedRunningTime="2026-07-15 23:43:26.063002696 +0000 UTC m=+795.351818126" Jul 15 23:43:26.100341 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.100297 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:43:26.100576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.100548 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" podUID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" containerName="authorino" containerID="cri-o://c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74" gracePeriod=30 Jul 15 23:43:26.363983 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.363948 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:43:26.438916 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.438874 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chm77\" (UniqueName: \"kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77\") pod \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " Jul 15 23:43:26.439115 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.438995 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert\") pod \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\" (UID: \"b89bcc97-cd48-4ccc-baf5-1ee26b688327\") " Jul 15 23:43:26.441369 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.441326 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77" (OuterVolumeSpecName: "kube-api-access-chm77") pod "b89bcc97-cd48-4ccc-baf5-1ee26b688327" (UID: "b89bcc97-cd48-4ccc-baf5-1ee26b688327"). InnerVolumeSpecName "kube-api-access-chm77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:43:26.450398 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.450360 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b89bcc97-cd48-4ccc-baf5-1ee26b688327" (UID: "b89bcc97-cd48-4ccc-baf5-1ee26b688327"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 23:43:26.539788 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.539743 2551 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b89bcc97-cd48-4ccc-baf5-1ee26b688327-tls-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:43:26.539788 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:26.539781 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chm77\" (UniqueName: \"kubernetes.io/projected/b89bcc97-cd48-4ccc-baf5-1ee26b688327-kube-api-access-chm77\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:43:27.051238 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.051203 2551 generic.go:356] "Generic (PLEG): container finished" podID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" containerID="c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74" exitCode=0 Jul 15 23:43:27.051730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.051259 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" Jul 15 23:43:27.051730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.051294 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" event={"ID":"b89bcc97-cd48-4ccc-baf5-1ee26b688327","Type":"ContainerDied","Data":"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74"} Jul 15 23:43:27.051730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.051342 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fb5b544f5-c5cvn" event={"ID":"b89bcc97-cd48-4ccc-baf5-1ee26b688327","Type":"ContainerDied","Data":"541d741028e650cb66ad21d3061c382066a82c263604213567b45df70cadca05"} Jul 15 23:43:27.051730 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.051368 2551 scope.go:117] "RemoveContainer" containerID="c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74" Jul 15 23:43:27.061762 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.061734 2551 scope.go:117] "RemoveContainer" containerID="c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74" Jul 15 23:43:27.062095 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:43:27.062078 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74\": container with ID starting with c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74 not found: ID does not exist" containerID="c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74" Jul 15 23:43:27.062191 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.062105 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74"} err="failed to get container status \"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74\": rpc error: code = NotFound desc = could not find container \"c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74\": container with ID starting with c203f5c8ea1db54c9c571f21752acc8f78e39ca563b3d0dd18d86fee30802d74 not found: ID does not exist" Jul 15 23:43:27.074983 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.074943 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:43:27.082596 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.082562 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fb5b544f5-c5cvn"] Jul 15 23:43:27.303652 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:27.303569 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" path="/var/lib/kubelet/pods/b89bcc97-cd48-4ccc-baf5-1ee26b688327/volumes" Jul 15 23:43:56.216552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:43:56.216505 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:44:05.912886 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:44:05.912850 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:44:17.218360 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:44:17.218319 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:44:25.620652 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:44:25.620604 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:44:35.720513 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:44:35.720467 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:45:44.215444 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:45:44.215403 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:45:52.318056 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:45:52.318015 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:03.216333 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:03.216299 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:12.720173 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:12.720135 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:23.518058 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:23.518019 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:33.818586 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:33.818493 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:45.919142 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:45.919088 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:46:56.424106 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:46:56.424063 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:47:17.227173 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:47:17.227102 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:47:29.823099 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:47:29.823057 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:47:53.127832 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:47:53.127775 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:48:04.420026 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:48:04.419938 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:48:20.719649 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:48:20.719608 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:48:30.917004 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:48:30.916965 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:48:41.122296 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:48:41.122255 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:48:50.913301 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:48:50.913263 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:49:01.020628 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:01.020584 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:49:24.859912 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:24.859859 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:49:24.860441 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:24.860120 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" podUID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" containerName="manager" containerID="cri-o://66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3" gracePeriod=10 Jul 15 23:49:28.420080 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.419996 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:49:28.573944 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.573890 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume\") pod \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " Jul 15 23:49:28.573944 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.573951 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzfg8\" (UniqueName: \"kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8\") pod \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\" (UID: \"2001ffa4-8784-46ff-9fa6-b5731cd921ad\") " Jul 15 23:49:28.574388 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.574364 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2001ffa4-8784-46ff-9fa6-b5731cd921ad" (UID: "2001ffa4-8784-46ff-9fa6-b5731cd921ad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:49:28.576153 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.576114 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8" (OuterVolumeSpecName: "kube-api-access-fzfg8") pod "2001ffa4-8784-46ff-9fa6-b5731cd921ad" (UID: "2001ffa4-8784-46ff-9fa6-b5731cd921ad"). InnerVolumeSpecName "kube-api-access-fzfg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:49:28.675466 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.675366 2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2001ffa4-8784-46ff-9fa6-b5731cd921ad-extensions-socket-volume\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:49:28.675466 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.675405 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfg8\" (UniqueName: \"kubernetes.io/projected/2001ffa4-8784-46ff-9fa6-b5731cd921ad-kube-api-access-fzfg8\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:49:28.911843 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.911810 2551 generic.go:356] "Generic (PLEG): container finished" podID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" containerID="66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3" exitCode=0 Jul 15 23:49:28.912022 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.911867 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" Jul 15 23:49:28.912022 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.911880 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" event={"ID":"2001ffa4-8784-46ff-9fa6-b5731cd921ad","Type":"ContainerDied","Data":"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3"} Jul 15 23:49:28.912022 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.911947 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q" event={"ID":"2001ffa4-8784-46ff-9fa6-b5731cd921ad","Type":"ContainerDied","Data":"efda183907d3865c37124e96587c0816a8ccff25d0bee856868358daae8679ea"} Jul 15 23:49:28.912022 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.911969 2551 scope.go:117] "RemoveContainer" containerID="66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3" Jul 15 23:49:28.922141 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.922106 2551 scope.go:117] "RemoveContainer" containerID="66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3" Jul 15 23:49:28.922483 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:49:28.922462 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3\": container with ID starting with 66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3 not found: ID does not exist" containerID="66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3" Jul 15 23:49:28.922547 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.922499 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3"} err="failed to get container status \"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3\": rpc error: code = NotFound desc = could not find container \"66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3\": container with ID starting with 66cf2f136fa4b4cfa8281ab416a08852e02a5270514e75c6440cf34cb905c2d3 not found: ID does not exist" Jul 15 23:49:28.933643 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.933557 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:49:28.938006 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:28.937977 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-d492q"] Jul 15 23:49:29.303338 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:49:29.303299 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" path="/var/lib/kubelet/pods/2001ffa4-8784-46ff-9fa6-b5731cd921ad/volumes" Jul 15 23:50:46.591367 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:50:46.591329 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:50:58.184225 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:50:58.184187 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:51:06.989279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:51:06.989185 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:51:18.796033 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:51:18.795985 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:51:29.583414 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:51:29.583378 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:51:39.681745 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:51:39.681698 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:51:49.797263 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:51:49.797218 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:00.186208 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:00.186172 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:10.478575 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:10.478536 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:18.682292 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:18.682250 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:27.582355 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:27.582315 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:34.991294 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:34.991208 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:52:54.383904 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:52:54.383866 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:03.894227 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:03.894194 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:15.280886 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:15.280849 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:24.582670 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:24.582633 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:30.887920 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:30.887865 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:40.188003 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:40.187967 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:48.083975 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:48.083937 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:53:56.889706 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:53:56.889669 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:06.800328 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:06.800248 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:15.593969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:15.593931 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:24.390540 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:24.390501 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:35.486041 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:35.485996 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:48.900665 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:48.900616 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:54:57.487227 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:54:57.487178 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:55:06.888144 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:55:06.888083 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:55:17.283107 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:55:17.283067 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:55:29.492011 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:55:29.491975 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:55:38.682747 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:55:38.682706 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:55:49.090813 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:55:49.090774 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:00.483201 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:00.483163 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:10.285640 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:10.285600 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:22.590516 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:22.590476 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:33.603021 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:33.602979 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:42.592354 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592321 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:56:42.592936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592804 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" containerName="authorino" Jul 15 23:56:42.592936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592824 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" containerName="authorino" Jul 15 23:56:42.592936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592851 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" containerName="manager" Jul 15 23:56:42.592936 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592861 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" containerName="manager" Jul 15 23:56:42.593158 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592951 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2001ffa4-8784-46ff-9fa6-b5731cd921ad" containerName="manager" Jul 15 23:56:42.593158 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.592964 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89bcc97-cd48-4ccc-baf5-1ee26b688327" containerName="authorino" Jul 15 23:56:42.596149 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.595795 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.598653 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.598619 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b1b98a6-openshift-default-dockercfg-7bcjc\"" Jul 15 23:56:42.598794 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.598689 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 23:56:42.617966 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.617926 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:56:42.632890 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.632852 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633200 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.632941 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633200 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633018 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633200 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633069 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw6s\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633200 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633095 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633200 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633195 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633494 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633237 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633494 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633259 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.633494 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.633297 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734456 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734409 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734480 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734509 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734540 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw6s\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734562 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734660 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734592 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734671 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734714 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734750 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734883 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.734923 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.734906 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.735245 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.735225 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.735304 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.735266 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.735481 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.735460 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.736952 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.736929 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.737098 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.737081 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.745247 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.745224 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.746023 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.746005 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw6s\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s\") pod \"e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:42.783957 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.783919 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:56:42.906884 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:42.906787 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:56:43.054609 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:43.054581 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:56:43.056572 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:56:43.056544 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861c2141_f42e_48de_adc7_b179bab7ce2e.slice/crio-3856cc82a04fafab4b76f9b38b5a3ee3428f7bc5c642751a1d671b6fa6ef8e54 WatchSource:0}: Error finding container 3856cc82a04fafab4b76f9b38b5a3ee3428f7bc5c642751a1d671b6fa6ef8e54: Status 404 returned error can't find the container with id 3856cc82a04fafab4b76f9b38b5a3ee3428f7bc5c642751a1d671b6fa6ef8e54 Jul 15 23:56:43.058671 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:43.058652 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 23:56:43.070255 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:43.070214 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" event={"ID":"861c2141-f42e-48de-adc7-b179bab7ce2e","Type":"ContainerStarted","Data":"3856cc82a04fafab4b76f9b38b5a3ee3428f7bc5c642751a1d671b6fa6ef8e54"} Jul 15 23:56:48.907490 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:56:48.907449 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:57:04.285494 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:04.285460 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:57:10.708091 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:10.708055 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:10.708401 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:10.708200 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:10.708401 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:10.708234 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:11.212781 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:11.212742 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" event={"ID":"861c2141-f42e-48de-adc7-b179bab7ce2e","Type":"ContainerStarted","Data":"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f"} Jul 15 23:57:11.213001 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:11.212864 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" podUID="861c2141-f42e-48de-adc7-b179bab7ce2e" containerName="istio-proxy" containerID="cri-o://5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f" gracePeriod=30 Jul 15 23:57:11.232192 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:11.232137 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" podStartSLOduration=1.58311255 podStartE2EDuration="29.232105086s" totalImagesPullingTime="27.648992536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:57:11.231883337 +0000 UTC m=+1620.520698766" watchObservedRunningTime="2026-07-15 23:57:11.232105086 +0000 UTC m=+1620.520920515" Jul 15 23:57:12.907478 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:12.907428 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:57:13.182465 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:13.182381 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:57:16.468618 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.468590 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:57:16.639685 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639648 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639728 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639749 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639786 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvw6s\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639806 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639835 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639866 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.639888 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639889 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.640279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.639955 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data\") pod \"861c2141-f42e-48de-adc7-b179bab7ce2e\" (UID: \"861c2141-f42e-48de-adc7-b179bab7ce2e\") " Jul 15 23:57:16.640279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640155 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:16.640279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640189 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:16.640279 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640244 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:57:16.640482 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640359 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:16.640482 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640372 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.640482 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640391 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.640482 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.640388 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:16.642007 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.641977 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 23:57:16.642115 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.642018 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:16.642115 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.642059 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:57:16.642115 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.642091 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s" (OuterVolumeSpecName: "kube-api-access-fvw6s") pod "861c2141-f42e-48de-adc7-b179bab7ce2e" (UID: "861c2141-f42e-48de-adc7-b179bab7ce2e"). InnerVolumeSpecName "kube-api-access-fvw6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:57:16.741450 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741407 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvw6s\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-kube-api-access-fvw6s\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741450 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741442 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741450 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741452 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741450 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741459 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741710 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741468 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741710 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741476 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/861c2141-f42e-48de-adc7-b179bab7ce2e-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:16.741710 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:16.741483 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/861c2141-f42e-48de-adc7-b179bab7ce2e-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:17.243666 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.243631 2551 generic.go:356] "Generic (PLEG): container finished" podID="861c2141-f42e-48de-adc7-b179bab7ce2e" containerID="5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f" exitCode=0 Jul 15 23:57:17.243839 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.243676 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" Jul 15 23:57:17.243839 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.243720 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" event={"ID":"861c2141-f42e-48de-adc7-b179bab7ce2e","Type":"ContainerDied","Data":"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f"} Jul 15 23:57:17.243839 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.243775 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4" event={"ID":"861c2141-f42e-48de-adc7-b179bab7ce2e","Type":"ContainerDied","Data":"3856cc82a04fafab4b76f9b38b5a3ee3428f7bc5c642751a1d671b6fa6ef8e54"} Jul 15 23:57:17.243839 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.243797 2551 scope.go:117] "RemoveContainer" containerID="5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f" Jul 15 23:57:17.253735 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.253715 2551 scope.go:117] "RemoveContainer" containerID="5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f" Jul 15 23:57:17.254012 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:57:17.253993 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f\": container with ID starting with 5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f not found: ID does not exist" containerID="5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f" Jul 15 23:57:17.254064 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.254020 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f"} err="failed to get container status \"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f\": rpc error: code = NotFound desc = could not find container \"5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f\": container with ID starting with 5ffbae3a25571bb0221cc309634347413c9bf4ac3b8ac60e39f848fae3e5ff8f not found: ID does not exist" Jul 15 23:57:17.265725 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.265690 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:57:17.270510 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.270481 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b1b98a6-openshift-default-56978746dc-pv2z4"] Jul 15 23:57:17.302355 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:17.302323 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="861c2141-f42e-48de-adc7-b179bab7ce2e" path="/var/lib/kubelet/pods/861c2141-f42e-48de-adc7-b179bab7ce2e/volumes" Jul 15 23:57:18.161507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.161468 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:18.165209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.162859 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="861c2141-f42e-48de-adc7-b179bab7ce2e" containerName="istio-proxy" Jul 15 23:57:18.165209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.162889 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="861c2141-f42e-48de-adc7-b179bab7ce2e" containerName="istio-proxy" Jul 15 23:57:18.165209 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.163069 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="861c2141-f42e-48de-adc7-b179bab7ce2e" containerName="istio-proxy" Jul 15 23:57:18.168598 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.168573 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.171744 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.171253 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-75c89f12-openshift-default-dockercfg-626wt\"" Jul 15 23:57:18.171744 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.171498 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 23:57:18.173576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.173542 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:18.250996 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.250960 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251016 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59m6b\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251047 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251107 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251161 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251198 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251278 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251275 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251720 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251302 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.251720 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.251325 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352439 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352480 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352480 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352513 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352593 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352621 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59m6b\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352646 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352703 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352673 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352709 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352743 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.352969 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.352960 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.353197 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.353174 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.353271 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.353231 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.353331 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.353268 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.353386 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.353371 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.355066 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.355044 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.355199 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.355184 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.366488 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.366456 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.367052 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.366980 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59m6b\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b\") pod \"e2e-derive-75c89f12-openshift-default-58795dd847-njhtm\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.481802 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.481701 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:18.626521 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.626492 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:18.628868 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:57:18.628841 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode084e11b_d171_4006_849c_69301b718159.slice/crio-ab08234676e034db16ab841af6dcc22b9c766c9a407d5e00d319742e15320620 WatchSource:0}: Error finding container ab08234676e034db16ab841af6dcc22b9c766c9a407d5e00d319742e15320620: Status 404 returned error can't find the container with id ab08234676e034db16ab841af6dcc22b9c766c9a407d5e00d319742e15320620 Jul 15 23:57:18.630869 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.630838 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:18.630926 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.630905 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:18.630965 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:18.630933 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:57:19.259076 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:19.259033 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" event={"ID":"e084e11b-d171-4006-849c-69301b718159","Type":"ContainerStarted","Data":"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603"} Jul 15 23:57:19.259483 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:19.259086 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" event={"ID":"e084e11b-d171-4006-849c-69301b718159","Type":"ContainerStarted","Data":"ab08234676e034db16ab841af6dcc22b9c766c9a407d5e00d319742e15320620"} Jul 15 23:57:19.278582 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:19.278515 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" podStartSLOduration=1.278495084 podStartE2EDuration="1.278495084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:57:19.277761169 +0000 UTC m=+1628.566576597" watchObservedRunningTime="2026-07-15 23:57:19.278495084 +0000 UTC m=+1628.567310517" Jul 15 23:57:19.482436 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:19.482401 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:19.487982 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:19.487953 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:20.264493 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:20.264459 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:20.265493 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:20.265470 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:21.586740 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:21.586648 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:57:23.825820 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:23.825779 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:23.826283 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:23.825988 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" podUID="e084e11b-d171-4006-849c-69301b718159" containerName="istio-proxy" containerID="cri-o://21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603" gracePeriod=30 Jul 15 23:57:29.992349 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:29.992311 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:30.060289 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060194 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060289 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060243 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060289 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060286 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060340 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060369 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060396 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060429 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59m6b\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060552 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060465 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.060785 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060639 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:30.060785 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060731 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:30.060785 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060765 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:57:30.060942 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.060813 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket\") pod \"e084e11b-d171-4006-849c-69301b718159\" (UID: \"e084e11b-d171-4006-849c-69301b718159\") " Jul 15 23:57:30.061152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.061066 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.061152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.061076 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:30.061152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.061087 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.061152 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.061102 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e084e11b-d171-4006-849c-69301b718159-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.061507 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.061485 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data" (OuterVolumeSpecName: "istio-data") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:30.062595 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.062571 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 23:57:30.062761 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.062739 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token" (OuterVolumeSpecName: "istio-token") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:57:30.062981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.062953 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:57:30.062981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.062955 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b" (OuterVolumeSpecName: "kube-api-access-59m6b") pod "e084e11b-d171-4006-849c-69301b718159" (UID: "e084e11b-d171-4006-849c-69301b718159"). InnerVolumeSpecName "kube-api-access-59m6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:57:30.161751 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161708 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.161751 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161740 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.161751 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161749 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.161751 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161758 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59m6b\" (UniqueName: \"kubernetes.io/projected/e084e11b-d171-4006-849c-69301b718159-kube-api-access-59m6b\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.162020 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161769 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e084e11b-d171-4006-849c-69301b718159-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.162020 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.161777 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e084e11b-d171-4006-849c-69301b718159-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:57:30.320887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.320786 2551 generic.go:356] "Generic (PLEG): container finished" podID="e084e11b-d171-4006-849c-69301b718159" containerID="21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603" exitCode=0 Jul 15 23:57:30.320887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.320826 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" event={"ID":"e084e11b-d171-4006-849c-69301b718159","Type":"ContainerDied","Data":"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603"} Jul 15 23:57:30.320887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.320844 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" Jul 15 23:57:30.320887 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.320868 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm" event={"ID":"e084e11b-d171-4006-849c-69301b718159","Type":"ContainerDied","Data":"ab08234676e034db16ab841af6dcc22b9c766c9a407d5e00d319742e15320620"} Jul 15 23:57:30.321228 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.320892 2551 scope.go:117] "RemoveContainer" containerID="21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603" Jul 15 23:57:30.330506 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.330486 2551 scope.go:117] "RemoveContainer" containerID="21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603" Jul 15 23:57:30.330844 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:57:30.330822 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603\": container with ID starting with 21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603 not found: ID does not exist" containerID="21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603" Jul 15 23:57:30.330902 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.330853 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603"} err="failed to get container status \"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603\": rpc error: code = NotFound desc = could not find container \"21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603\": container with ID starting with 21388bf601df83375f66bd59d08bbab5d59c14f721a48b5270352554ac280603 not found: ID does not exist" Jul 15 23:57:30.360838 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.360750 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:30.363383 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:30.363305 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-75c89f12-openshift-default-58795dd847-njhtm"] Jul 15 23:57:31.303485 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:31.303453 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e084e11b-d171-4006-849c-69301b718159" path="/var/lib/kubelet/pods/e084e11b-d171-4006-849c-69301b718159/volumes" Jul 15 23:57:31.591527 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:57:31.591430 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:58:58.381576 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:58:58.381531 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:59:07.395772 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:07.395733 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:59:10.190607 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.190570 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:10.190981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.190909 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e084e11b-d171-4006-849c-69301b718159" containerName="istio-proxy" Jul 15 23:59:10.190981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.190920 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e084e11b-d171-4006-849c-69301b718159" containerName="istio-proxy" Jul 15 23:59:10.190981 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.190976 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e084e11b-d171-4006-849c-69301b718159" containerName="istio-proxy" Jul 15 23:59:10.194086 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.194065 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.196587 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.196561 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 23:59:10.196587 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.196577 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f5988523-openshift-default-dockercfg-jg6r2\"" Jul 15 23:59:10.217648 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.217617 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:10.324632 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324591 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324632 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324638 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf6q\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324729 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324759 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324780 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324822 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324842 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324867 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.324913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.324886 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.425835 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425787 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425839 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425877 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425903 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425961 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.425987 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf6q\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426048 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426037 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426060 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426087 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426288 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426429 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426386 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426600 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426489 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426675 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426656 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.426766 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.426746 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.428300 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.428280 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.428594 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.428576 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.434273 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.434252 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf6q\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.434574 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.434558 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token\") pod \"e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.505246 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.505213 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:10.655147 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.655038 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:10.657320 ip-10-0-142-149 kubenswrapper[2551]: W0715 23:59:10.657286 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc08453_8f69_4a14_866e_d1a903616bef.slice/crio-49778cbb2289c6523321d3098fbd0d3d1205ec96377c87f9d23543a2f2a87301 WatchSource:0}: Error finding container 49778cbb2289c6523321d3098fbd0d3d1205ec96377c87f9d23543a2f2a87301: Status 404 returned error can't find the container with id 49778cbb2289c6523321d3098fbd0d3d1205ec96377c87f9d23543a2f2a87301 Jul 15 23:59:10.659420 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.659385 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:10.659484 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.659453 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:10.659534 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.659481 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:10.818120 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.817955 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerStarted","Data":"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b"} Jul 15 23:59:10.818120 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.818024 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerStarted","Data":"49778cbb2289c6523321d3098fbd0d3d1205ec96377c87f9d23543a2f2a87301"} Jul 15 23:59:10.841458 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:10.841401 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podStartSLOduration=0.841385491 podStartE2EDuration="841.385491ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 23:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 23:59:10.838927632 +0000 UTC m=+1740.127743093" watchObservedRunningTime="2026-07-15 23:59:10.841385491 +0000 UTC m=+1740.130200919" Jul 15 23:59:11.505989 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:11.505957 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:11.510760 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:11.510736 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:11.823508 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:11.823428 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:11.824495 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:11.824474 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:17.798904 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:17.798868 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:59:26.825195 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.825155 2551 patch_prober.go:28] interesting pod/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 23:59:26.825636 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.825214 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 23:59:26.909739 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.909705 2551 generic.go:356] "Generic (PLEG): container finished" podID="2cc08453-8f69-4a14-866e-d1a903616bef" containerID="f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b" exitCode=0 Jul 15 23:59:26.909901 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.909774 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerDied","Data":"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b"} Jul 15 23:59:26.910198 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.910184 2551 scope.go:117] "RemoveContainer" containerID="f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b" Jul 15 23:59:26.911231 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.911200 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:26.911337 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.911277 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:26.911337 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:26.911311 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 23:59:27.916946 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:27.916910 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerStarted","Data":"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11"} Jul 15 23:59:28.505893 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:28.505850 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:28.506116 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:28.506095 2551 patch_prober.go:28] interesting pod/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 23:59:28.506208 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:28.506172 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 23:59:29.506388 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:29.506349 2551 patch_prober.go:28] interesting pod/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 23:59:29.506868 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:29.506422 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 23:59:30.505638 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:30.505592 2551 patch_prober.go:28] interesting pod/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 23:59:30.505846 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:30.505663 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 23:59:31.505869 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:31.505831 2551 patch_prober.go:28] interesting pod/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 23:59:31.506302 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:31.505925 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 23:59:32.509713 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:32.509678 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:32.510235 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:32.510149 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:32.510913 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:32.510895 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:33.682786 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:33.682748 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 15 23:59:36.967825 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:36.967793 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:36.968316 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:36.968025 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" containerID="cri-o://e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11" gracePeriod=30 Jul 15 23:59:42.225367 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.225341 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:42.311708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311671 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311708 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311720 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311749 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311770 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311805 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311830 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311868 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.311984 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.311895 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.312309 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312052 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:59:42.312309 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312071 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:59:42.312309 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312112 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:59:42.312461 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312401 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 23:59:42.312593 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312562 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnf6q\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q\") pod \"2cc08453-8f69-4a14-866e-d1a903616bef\" (UID: \"2cc08453-8f69-4a14-866e-d1a903616bef\") " Jul 15 23:59:42.312955 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312931 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.313184 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.313116 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.313184 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.313176 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.313361 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.313191 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cc08453-8f69-4a14-866e-d1a903616bef-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.313361 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.312934 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data" (OuterVolumeSpecName: "istio-data") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:59:42.314318 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.314291 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token" (OuterVolumeSpecName: "istio-token") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:59:42.314413 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.314311 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 23:59:42.314413 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.314405 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 23:59:42.314750 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.314722 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q" (OuterVolumeSpecName: "kube-api-access-fnf6q") pod "2cc08453-8f69-4a14-866e-d1a903616bef" (UID: "2cc08453-8f69-4a14-866e-d1a903616bef"). InnerVolumeSpecName "kube-api-access-fnf6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 23:59:42.414230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.414193 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.414230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.414225 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.414230 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.414238 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cc08453-8f69-4a14-866e-d1a903616bef-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.414468 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.414247 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cc08453-8f69-4a14-866e-d1a903616bef-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.414468 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.414256 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnf6q\" (UniqueName: \"kubernetes.io/projected/2cc08453-8f69-4a14-866e-d1a903616bef-kube-api-access-fnf6q\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 15 23:59:42.998706 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.998672 2551 generic.go:356] "Generic (PLEG): container finished" podID="2cc08453-8f69-4a14-866e-d1a903616bef" containerID="e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11" exitCode=0 Jul 15 23:59:42.998860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.998719 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" Jul 15 23:59:42.998860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.998752 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerDied","Data":"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11"} Jul 15 23:59:42.998860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.998805 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z" event={"ID":"2cc08453-8f69-4a14-866e-d1a903616bef","Type":"ContainerDied","Data":"49778cbb2289c6523321d3098fbd0d3d1205ec96377c87f9d23543a2f2a87301"} Jul 15 23:59:42.998860 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:42.998827 2551 scope.go:117] "RemoveContainer" containerID="e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11" Jul 15 23:59:43.008468 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.008449 2551 scope.go:117] "RemoveContainer" containerID="f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b" Jul 15 23:59:43.016744 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.016724 2551 scope.go:117] "RemoveContainer" containerID="e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11" Jul 15 23:59:43.017034 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:59:43.017015 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11\": container with ID starting with e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11 not found: ID does not exist" containerID="e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11" Jul 15 23:59:43.017077 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.017043 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11"} err="failed to get container status \"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11\": rpc error: code = NotFound desc = could not find container \"e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11\": container with ID starting with e9ccfbf44970d12d9136439bbfa9c8aed4bec8e2c5d3f758ddfb61ad368f5d11 not found: ID does not exist" Jul 15 23:59:43.017077 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.017063 2551 scope.go:117] "RemoveContainer" containerID="f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b" Jul 15 23:59:43.017332 ip-10-0-142-149 kubenswrapper[2551]: E0715 23:59:43.017316 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b\": container with ID starting with f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b not found: ID does not exist" containerID="f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b" Jul 15 23:59:43.017379 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.017338 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b"} err="failed to get container status \"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b\": rpc error: code = NotFound desc = could not find container \"f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b\": container with ID starting with f7b726a5351f94cc106c299ddf5e0e7c3de14cc9a3e106c7d83ecb9b699c336b not found: ID does not exist" Jul 15 23:59:43.020702 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.020668 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:43.026255 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.026229 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f5988523-openshift-default-586bf95c95-7bd6z"] Jul 15 23:59:43.302927 ip-10-0-142-149 kubenswrapper[2551]: I0715 23:59:43.302890 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" path="/var/lib/kubelet/pods/2cc08453-8f69-4a14-866e-d1a903616bef/volumes" Jul 16 00:00:00.142603 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142564 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29736000-m2s58"] Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142883 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142894 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142904 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142910 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142972 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.143099 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.142981 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc08453-8f69-4a14-866e-d1a903616bef" containerName="istio-proxy" Jul 16 00:00:00.145690 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.145674 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.148358 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.148330 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-6648n\"" Jul 16 00:00:00.148507 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.148330 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 16 00:00:00.157285 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.157251 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29736000-m2s58"] Jul 16 00:00:00.274214 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.274169 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhfn\" (UniqueName: \"kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.274408 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.274297 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.375525 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.375483 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.375735 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.375546 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhfn\" (UniqueName: \"kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.376235 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.376209 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.383463 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.383432 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhfn\" (UniqueName: \"kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn\") pod \"image-pruner-29736000-m2s58\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.473746 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.473646 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:00.613635 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:00.613605 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29736000-m2s58"] Jul 16 00:00:00.617407 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:00:00.617373 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5075908_2d0e_4db7_b2b2_6ff86b85e549.slice/crio-a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487 WatchSource:0}: Error finding container a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487: Status 404 returned error can't find the container with id a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487 Jul 16 00:00:01.092994 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:01.092959 2551 generic.go:356] "Generic (PLEG): container finished" podID="f5075908-2d0e-4db7-b2b2-6ff86b85e549" containerID="a33defddbb734856ba59a334364a16b9826620f2b810224d65044c392e867423" exitCode=0 Jul 16 00:00:01.093203 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:01.093029 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29736000-m2s58" event={"ID":"f5075908-2d0e-4db7-b2b2-6ff86b85e549","Type":"ContainerDied","Data":"a33defddbb734856ba59a334364a16b9826620f2b810224d65044c392e867423"} Jul 16 00:00:01.093203 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:01.093079 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29736000-m2s58" event={"ID":"f5075908-2d0e-4db7-b2b2-6ff86b85e549","Type":"ContainerStarted","Data":"a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487"} Jul 16 00:00:01.293399 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:01.293354 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:00:02.242051 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.242016 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:02.393288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.393192 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca\") pod \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " Jul 16 00:00:02.393288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.393241 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdhfn\" (UniqueName: \"kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn\") pod \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\" (UID: \"f5075908-2d0e-4db7-b2b2-6ff86b85e549\") " Jul 16 00:00:02.393778 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.393543 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca" (OuterVolumeSpecName: "serviceca") pod "f5075908-2d0e-4db7-b2b2-6ff86b85e549" (UID: "f5075908-2d0e-4db7-b2b2-6ff86b85e549"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:00:02.395529 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.395492 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn" (OuterVolumeSpecName: "kube-api-access-jdhfn") pod "f5075908-2d0e-4db7-b2b2-6ff86b85e549" (UID: "f5075908-2d0e-4db7-b2b2-6ff86b85e549"). InnerVolumeSpecName "kube-api-access-jdhfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:00:02.494198 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.494099 2551 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f5075908-2d0e-4db7-b2b2-6ff86b85e549-serviceca\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:00:02.494198 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:02.494144 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhfn\" (UniqueName: \"kubernetes.io/projected/f5075908-2d0e-4db7-b2b2-6ff86b85e549-kube-api-access-jdhfn\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:00:03.105639 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:03.105598 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29736000-m2s58" Jul 16 00:00:03.105865 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:03.105607 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29736000-m2s58" event={"ID":"f5075908-2d0e-4db7-b2b2-6ff86b85e549","Type":"ContainerDied","Data":"a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487"} Jul 16 00:00:03.105865 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:03.105733 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a927b7c97db30fd1a37faece0e76918fcda3605773046a53ddc2f019369be487" Jul 16 00:00:13.987668 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:13.987624 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:00:50.322202 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.322168 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:00:50.322670 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.322494 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5075908-2d0e-4db7-b2b2-6ff86b85e549" containerName="image-pruner" Jul 16 00:00:50.322670 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.322504 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5075908-2d0e-4db7-b2b2-6ff86b85e549" containerName="image-pruner" Jul 16 00:00:50.322670 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.322578 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5075908-2d0e-4db7-b2b2-6ff86b85e549" containerName="image-pruner" Jul 16 00:00:50.325753 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.325725 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.328048 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.328028 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9d3c28-openshift-default-dockercfg-cdprw\"" Jul 16 00:00:50.328213 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.328195 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 00:00:50.337642 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.337612 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:00:50.402836 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402790 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.402836 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402835 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403071 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402896 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403071 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402959 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403071 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402978 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403071 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.402999 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403239 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.403116 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403239 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.403188 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.403426 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.403407 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcm8\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504501 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504456 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504501 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504506 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504528 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504555 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504572 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504592 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504641 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504685 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.504798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504733 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcm8\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.505178 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504988 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.505178 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.504987 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.505262 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.505199 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.505304 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.505276 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.505410 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.505381 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.507312 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.507282 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.507534 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.507512 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.512782 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.512741 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.512955 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.512930 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcm8\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8\") pod \"e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.637572 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.637439 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:50.786597 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.786570 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:00:50.789355 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:00:50.789327 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac45dc55_a736_4122_a421_922232cb9ecc.slice/crio-9d78e3188e9a808df3170366ab12d56b0213dee214193546a02a049816b5ab7b WatchSource:0}: Error finding container 9d78e3188e9a808df3170366ab12d56b0213dee214193546a02a049816b5ab7b: Status 404 returned error can't find the container with id 9d78e3188e9a808df3170366ab12d56b0213dee214193546a02a049816b5ab7b Jul 16 00:00:50.791598 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.791566 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:50.791693 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.791636 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:50.791693 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:50.791664 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:51.343585 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:51.343543 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerStarted","Data":"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad"} Jul 16 00:00:51.344005 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:51.343607 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerStarted","Data":"9d78e3188e9a808df3170366ab12d56b0213dee214193546a02a049816b5ab7b"} Jul 16 00:00:51.364622 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:51.364555 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podStartSLOduration=1.364531542 podStartE2EDuration="1.364531542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:00:51.362777348 +0000 UTC m=+1840.651592778" watchObservedRunningTime="2026-07-16 00:00:51.364531542 +0000 UTC m=+1840.653346970" Jul 16 00:00:51.644159 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:51.637921 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:51.648505 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:51.648479 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:52.349021 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:52.348978 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:52.350118 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:52.350090 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:00:58.516286 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.516242 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:00:58.519870 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.519844 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.527330 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.522987 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2a1132-openshift-default-dockercfg-rdbv4\"" Jul 16 00:00:58.540705 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.540677 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:00:58.678003 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.677963 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678201 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678019 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678201 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678073 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678201 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678120 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678340 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678195 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqrf\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678340 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678266 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678340 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678314 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678440 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678361 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.678440 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.678396 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779283 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779323 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779347 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779364 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqrf\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779710 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779408 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779710 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779453 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779710 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779590 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779710 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779668 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779912 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779719 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.779912 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779728 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.780001 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.779929 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.780138 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.780093 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.780214 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.780169 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.780266 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.780257 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.782288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.782262 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.782679 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.782656 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.788175 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.788141 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqrf\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.788885 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.788861 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token\") pod \"e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.837945 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.837900 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:00:58.994004 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.993977 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:00:58.996137 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:00:58.996086 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0534ffd_4c8b_4992_861e_29f84a819896.slice/crio-f241c88a5c4c00c1c09b3b19c86dd43d27d45565cb20d0d934c2ec3ca7e9c4b0 WatchSource:0}: Error finding container f241c88a5c4c00c1c09b3b19c86dd43d27d45565cb20d0d934c2ec3ca7e9c4b0: Status 404 returned error can't find the container with id f241c88a5c4c00c1c09b3b19c86dd43d27d45565cb20d0d934c2ec3ca7e9c4b0 Jul 16 00:00:58.998305 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.998266 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:58.998433 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.998355 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:58.998433 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:58.998396 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:00:59.385557 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:59.385465 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerStarted","Data":"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806"} Jul 16 00:00:59.385557 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:59.385527 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerStarted","Data":"f241c88a5c4c00c1c09b3b19c86dd43d27d45565cb20d0d934c2ec3ca7e9c4b0"} Jul 16 00:00:59.404605 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:59.404554 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podStartSLOduration=1.404535041 podStartE2EDuration="1.404535041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:00:59.403077173 +0000 UTC m=+1848.691892607" watchObservedRunningTime="2026-07-16 00:00:59.404535041 +0000 UTC m=+1848.693350532" Jul 16 00:00:59.838225 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:00:59.838187 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:00.314656 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:00.314618 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:01:00.843426 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:00.843388 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:01.395983 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:01.395954 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:01.396987 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:01.396966 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:10.902439 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:10.902391 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:01:12.458275 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.458231 2551 generic.go:356] "Generic (PLEG): container finished" podID="ac45dc55-a736-4122-a421-922232cb9ecc" containerID="d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad" exitCode=0 Jul 16 00:01:12.458707 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.458304 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerDied","Data":"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad"} Jul 16 00:01:12.458751 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.458729 2551 scope.go:117] "RemoveContainer" containerID="d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad" Jul 16 00:01:12.459932 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.459901 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:12.460006 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.459978 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:12.460046 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:12.460010 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:13.492828 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:13.492778 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerStarted","Data":"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0"} Jul 16 00:01:13.638368 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:13.638327 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:01:13.638556 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:13.638530 2551 patch_prober.go:28] interesting pod/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:01:13.638618 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:13.638596 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:01:14.552167 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.552107 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:01:14.554601 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.554578 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.558539 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.558508 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9d3c28\"/\"kube-root-ca.crt\"" Jul 16 00:01:14.558712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.558539 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9d3c28\"/\"test-model-9d3c28-kserve-self-signed-certs\"" Jul 16 00:01:14.558712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.558554 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9d3c28\"/\"openshift-service-ca.crt\"" Jul 16 00:01:14.558712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.558558 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9d3c28\"/\"default-dockercfg-w7mbt\"" Jul 16 00:01:14.583315 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.583062 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:01:14.626289 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626244 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.626289 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626290 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgb8h\" (UniqueName: \"kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.626509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626313 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.626509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626441 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.626582 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626500 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.626582 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.626536 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.638374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.638336 2551 patch_prober.go:28] interesting pod/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:01:14.638555 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.638410 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:01:14.727677 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727624 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.727677 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727665 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgb8h\" (UniqueName: \"kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.727677 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727685 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728016 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727728 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728016 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727747 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728016 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.727765 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728242 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.728165 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728242 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.728204 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.728362 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.728287 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.730425 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.730379 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.730643 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.730618 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.736780 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.736739 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgb8h\" (UniqueName: \"kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h\") pod \"test-model-9d3c28-kserve-68898d7f95-ctklf\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:14.898425 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:14.898323 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:15.087169 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.086386 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:01:15.094415 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:01:15.094376 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9fd72d7_4efa_479c_b8c2_8c9fe28e9360.slice/crio-9a31bd5c25607b940b8fc87ac6f0a34cf5b7727937880726fe7984aec40a2113 WatchSource:0}: Error finding container 9a31bd5c25607b940b8fc87ac6f0a34cf5b7727937880726fe7984aec40a2113: Status 404 returned error can't find the container with id 9a31bd5c25607b940b8fc87ac6f0a34cf5b7727937880726fe7984aec40a2113 Jul 16 00:01:15.504530 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.504432 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" event={"ID":"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360","Type":"ContainerStarted","Data":"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80"} Jul 16 00:01:15.504530 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.504481 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" event={"ID":"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360","Type":"ContainerStarted","Data":"9a31bd5c25607b940b8fc87ac6f0a34cf5b7727937880726fe7984aec40a2113"} Jul 16 00:01:15.522975 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.522912 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" podStartSLOduration=1.522894687 podStartE2EDuration="1.522894687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:01:15.520909949 +0000 UTC m=+1864.809725530" watchObservedRunningTime="2026-07-16 00:01:15.522894687 +0000 UTC m=+1864.811710116" Jul 16 00:01:15.638077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.638035 2551 patch_prober.go:28] interesting pod/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:01:15.638489 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:15.638103 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:01:16.397327 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.397284 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:16.397511 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.397344 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:16.510751 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.510717 2551 generic.go:356] "Generic (PLEG): container finished" podID="d0534ffd-4c8b-4992-861e-29f84a819896" containerID="2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806" exitCode=0 Jul 16 00:01:16.510958 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.510795 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerDied","Data":"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806"} Jul 16 00:01:16.511360 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.511339 2551 scope.go:117] "RemoveContainer" containerID="2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806" Jul 16 00:01:16.512474 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.512441 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:16.512568 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.512522 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:16.512568 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.512559 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:01:16.638438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.638396 2551 patch_prober.go:28] interesting pod/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 16 00:01:16.638830 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:16.638466 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 16 00:01:17.517832 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.517793 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerStarted","Data":"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79"} Jul 16 00:01:17.642769 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.642735 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:01:17.643285 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.643027 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:01:17.643770 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.643752 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:01:17.838673 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.838577 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:17.838871 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.838810 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:17.838998 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:17.838865 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:18.838491 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:18.838449 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:18.838896 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:18.838506 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:19.710844 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:19.710784 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:01:19.838582 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:19.838536 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:19.839092 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:19.838615 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:20.838743 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:20.838704 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:20.839286 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:20.839256 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:21.838379 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:21.838342 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 16 00:01:21.838654 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:21.838403 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 16 00:01:22.842505 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:22.842472 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:22.842914 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:22.842759 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:22.843495 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:22.843479 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:01:24.898424 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:24.898388 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:24.898843 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:24.898439 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:24.911809 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:24.911776 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:25.572258 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:25.572229 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:01:36.723839 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:36.723729 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:01:48.923262 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:48.923221 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:01:54.487511 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:01:54.487470 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:02:03.946022 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:03.945957 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" podUID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" containerName="main" containerID="cri-o://02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80" gracePeriod=45 Jul 16 00:02:04.200643 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.200568 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:02:04.293059 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293019 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgb8h\" (UniqueName: \"kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293339 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293151 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293339 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293183 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293339 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293245 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293339 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293281 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293339 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293326 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home\") pod \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\" (UID: \"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360\") " Jul 16 00:02:04.293640 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293554 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:04.293640 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293570 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:04.293640 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.293615 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home" (OuterVolumeSpecName: "home") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:04.295569 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.295537 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:02:04.295569 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.295551 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h" (OuterVolumeSpecName: "kube-api-access-cgb8h") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "kube-api-access-cgb8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:02:04.295569 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.295541 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm" (OuterVolumeSpecName: "dshm") pod "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" (UID: "d9fd72d7-4efa-479c-b8c2-8c9fe28e9360"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:04.394438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394395 2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-model-cache\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.394438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394435 2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-home\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.394659 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394447 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgb8h\" (UniqueName: \"kubernetes.io/projected/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-kube-api-access-cgb8h\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.394659 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394462 2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tls-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.394659 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394474 2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-dshm\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.394659 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.394485 2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360-tmp-dir\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:04.769646 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.769613 2551 generic.go:356] "Generic (PLEG): container finished" podID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" containerID="02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80" exitCode=0 Jul 16 00:02:04.769845 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.769655 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" Jul 16 00:02:04.769845 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.769666 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" event={"ID":"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360","Type":"ContainerDied","Data":"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80"} Jul 16 00:02:04.769845 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.769705 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf" event={"ID":"d9fd72d7-4efa-479c-b8c2-8c9fe28e9360","Type":"ContainerDied","Data":"9a31bd5c25607b940b8fc87ac6f0a34cf5b7727937880726fe7984aec40a2113"} Jul 16 00:02:04.769845 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.769724 2551 scope.go:117] "RemoveContainer" containerID="02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80" Jul 16 00:02:04.780373 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.780346 2551 scope.go:117] "RemoveContainer" containerID="02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80" Jul 16 00:02:04.780739 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:02:04.780714 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80\": container with ID starting with 02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80 not found: ID does not exist" containerID="02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80" Jul 16 00:02:04.780823 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.780766 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80"} err="failed to get container status \"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80\": rpc error: code = NotFound desc = could not find container \"02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80\": container with ID starting with 02afa2d3af6f33675adb842e11937b9557aea47cdcc1db7c989c08d415f94d80 not found: ID does not exist" Jul 16 00:02:04.794923 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.794875 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:02:04.798396 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:04.798352 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9d3c28/test-model-9d3c28-kserve-68898d7f95-ctklf"] Jul 16 00:02:05.308878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:05.308835 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" path="/var/lib/kubelet/pods/d9fd72d7-4efa-479c-b8c2-8c9fe28e9360/volumes" Jul 16 00:02:35.291658 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:35.291621 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:02:37.906715 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:37.906681 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:02:37.907204 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:37.906918 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" containerID="cri-o://d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0" gracePeriod=30 Jul 16 00:02:43.164413 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.164381 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:02:43.248063 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.247959 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248063 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248034 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248063 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248055 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248072 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdcm8\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248097 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248119 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248202 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248252 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248279 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token\") pod \"ac45dc55-a736-4122-a421-922232cb9ecc\" (UID: \"ac45dc55-a736-4122-a421-922232cb9ecc\") " Jul 16 00:02:43.248656 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248454 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:43.248656 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248606 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.248656 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248637 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:02:43.248824 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248782 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:43.248824 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.248770 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:43.249451 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.249417 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:43.250687 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.250659 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:02:43.250687 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.250672 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:43.250687 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.250660 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8" (OuterVolumeSpecName: "kube-api-access-cdcm8") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "kube-api-access-cdcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:02:43.251414 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.251393 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac45dc55-a736-4122-a421-922232cb9ecc" (UID: "ac45dc55-a736-4122-a421-922232cb9ecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:02:43.349207 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349171 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdcm8\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-kube-api-access-cdcm8\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349207 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349204 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349207 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349213 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349207 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349223 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349207 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349232 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac45dc55-a736-4122-a421-922232cb9ecc-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349548 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349243 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac45dc55-a736-4122-a421-922232cb9ecc-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349548 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349255 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac45dc55-a736-4122-a421-922232cb9ecc-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.349548 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.349264 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac45dc55-a736-4122-a421-922232cb9ecc-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:43.981742 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.981706 2551 generic.go:356] "Generic (PLEG): container finished" podID="ac45dc55-a736-4122-a421-922232cb9ecc" containerID="d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0" exitCode=0 Jul 16 00:02:43.981960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.981758 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" Jul 16 00:02:43.981960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.981797 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerDied","Data":"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0"} Jul 16 00:02:43.981960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.981855 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r" event={"ID":"ac45dc55-a736-4122-a421-922232cb9ecc","Type":"ContainerDied","Data":"9d78e3188e9a808df3170366ab12d56b0213dee214193546a02a049816b5ab7b"} Jul 16 00:02:43.981960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.981880 2551 scope.go:117] "RemoveContainer" containerID="d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0" Jul 16 00:02:43.992880 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:43.992850 2551 scope.go:117] "RemoveContainer" containerID="d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad" Jul 16 00:02:44.002048 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.002024 2551 scope.go:117] "RemoveContainer" containerID="d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0" Jul 16 00:02:44.002393 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:02:44.002374 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0\": container with ID starting with d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0 not found: ID does not exist" containerID="d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0" Jul 16 00:02:44.002481 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.002407 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0"} err="failed to get container status \"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0\": rpc error: code = NotFound desc = could not find container \"d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0\": container with ID starting with d643fd0e12e64db474c68346f89e2609489f0e29b6525debb745f7e5cf406ba0 not found: ID does not exist" Jul 16 00:02:44.002481 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.002434 2551 scope.go:117] "RemoveContainer" containerID="d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad" Jul 16 00:02:44.002704 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:02:44.002684 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad\": container with ID starting with d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad not found: ID does not exist" containerID="d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad" Jul 16 00:02:44.002747 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.002713 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad"} err="failed to get container status \"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad\": rpc error: code = NotFound desc = could not find container \"d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad\": container with ID starting with d3c8f9f70733613ffb35bf885a1bbf57920669f2d1dc66477116913015ac51ad not found: ID does not exist" Jul 16 00:02:44.003631 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.003611 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:02:44.007485 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:44.007453 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9d3c28-openshift-default-54f74b4f79-x6b9r"] Jul 16 00:02:45.302616 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:45.302581 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" path="/var/lib/kubelet/pods/ac45dc55-a736-4122-a421-922232cb9ecc/volumes" Jul 16 00:02:47.985800 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:47.985760 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:02:50.946450 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:50.946406 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:02:50.946941 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:50.946636 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" containerID="cri-o://d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79" gracePeriod=30 Jul 16 00:02:51.851960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.851918 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:02:51.852709 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852682 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:02:51.852709 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852708 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852734 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" containerName="main" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852744 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" containerName="main" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852755 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852764 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852833 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9fd72d7-4efa-479c-b8c2-8c9fe28e9360" containerName="main" Jul 16 00:02:51.852878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.852845 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:02:51.855253 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.855228 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.861439 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.861398 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3cb745-openshift-default-dockercfg-zqrfd\"" Jul 16 00:02:51.871098 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.871066 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:02:51.920041 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.919997 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920041 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920043 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920073 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2jp\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920192 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920241 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920279 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920295 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920357 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920318 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:51.920671 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:51.920360 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.020924 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.020884 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.020932 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.020964 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.020996 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021034 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021152 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021198 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2jp\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021258 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021453 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021301 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021839 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021659 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021839 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021746 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021839 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021785 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.021986 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.021876 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.022248 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.022223 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.023759 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.023733 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.024261 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.024237 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.029541 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.029504 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.029697 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.029666 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2jp\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp\") pod \"e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.176771 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.176678 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:52.326845 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.326820 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:02:52.329220 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:02:52.329193 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7069e248_241a_4eba_b5d9_a89601edfffd.slice/crio-ebe058abaa7c78a1737f8b698c349ca1a34c882064e203b5b7a08fc133496af5 WatchSource:0}: Error finding container ebe058abaa7c78a1737f8b698c349ca1a34c882064e203b5b7a08fc133496af5: Status 404 returned error can't find the container with id ebe058abaa7c78a1737f8b698c349ca1a34c882064e203b5b7a08fc133496af5 Jul 16 00:02:52.331866 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.331829 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:02:52.331956 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.331902 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:02:52.331956 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.331934 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:02:52.845582 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.845519 2551 patch_prober.go:28] interesting pod/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:02:52.845782 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:52.845608 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:02:53.032696 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:53.032659 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerStarted","Data":"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd"} Jul 16 00:02:53.033102 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:53.032726 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerStarted","Data":"ebe058abaa7c78a1737f8b698c349ca1a34c882064e203b5b7a08fc133496af5"} Jul 16 00:02:53.054308 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:53.054249 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podStartSLOduration=2.054231655 podStartE2EDuration="2.054231655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:02:53.051760061 +0000 UTC m=+1962.340575490" watchObservedRunningTime="2026-07-16 00:02:53.054231655 +0000 UTC m=+1962.343047084" Jul 16 00:02:53.176966 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:53.176881 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:53.182064 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:53.182036 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:54.038458 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:54.038420 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:54.039607 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:54.039583 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:02:56.216471 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.216387 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:02:56.260901 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.260852 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261095 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.260910 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261095 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.260948 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261095 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261042 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261095 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261076 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qqrf\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261369 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261114 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261431 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261353 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:56.261494 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261430 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261549 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261507 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261604 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261582 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs\") pod \"d0534ffd-4c8b-4992-861e-29f84a819896\" (UID: \"d0534ffd-4c8b-4992-861e-29f84a819896\") " Jul 16 00:02:56.261890 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261861 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:56.261990 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.261908 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:02:56.262052 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.262011 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0534ffd-4c8b-4992-861e-29f84a819896-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.262052 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.262029 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.262052 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.262043 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.262228 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.262176 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:56.262367 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.262340 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:56.263781 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.263734 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:02:56.264022 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.263933 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf" (OuterVolumeSpecName: "kube-api-access-4qqrf") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "kube-api-access-4qqrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:02:56.264282 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.264252 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:02:56.264386 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.264340 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0534ffd-4c8b-4992-861e-29f84a819896" (UID: "d0534ffd-4c8b-4992-861e-29f84a819896"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:02:56.363387 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363341 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqrf\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-kube-api-access-4qqrf\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.363387 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363377 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.363387 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363388 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0534ffd-4c8b-4992-861e-29f84a819896-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.363387 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363396 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.363672 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363404 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0534ffd-4c8b-4992-861e-29f84a819896-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:56.363672 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:56.363412 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0534ffd-4c8b-4992-861e-29f84a819896-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:02:57.055970 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.055933 2551 generic.go:356] "Generic (PLEG): container finished" podID="d0534ffd-4c8b-4992-861e-29f84a819896" containerID="d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79" exitCode=0 Jul 16 00:02:57.056175 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.055997 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" Jul 16 00:02:57.056175 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.056013 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerDied","Data":"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79"} Jul 16 00:02:57.056175 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.056090 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt" event={"ID":"d0534ffd-4c8b-4992-861e-29f84a819896","Type":"ContainerDied","Data":"f241c88a5c4c00c1c09b3b19c86dd43d27d45565cb20d0d934c2ec3ca7e9c4b0"} Jul 16 00:02:57.056175 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.056112 2551 scope.go:117] "RemoveContainer" containerID="d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79" Jul 16 00:02:57.068725 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.068563 2551 scope.go:117] "RemoveContainer" containerID="2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806" Jul 16 00:02:57.078041 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.078015 2551 scope.go:117] "RemoveContainer" containerID="d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79" Jul 16 00:02:57.078438 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:02:57.078414 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79\": container with ID starting with d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79 not found: ID does not exist" containerID="d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79" Jul 16 00:02:57.078516 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.078446 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79"} err="failed to get container status \"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79\": rpc error: code = NotFound desc = could not find container \"d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79\": container with ID starting with d1d0af7538bfeb4120ff22a598d5ad18952381eff25971843bb36019a5fc8f79 not found: ID does not exist" Jul 16 00:02:57.078516 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.078468 2551 scope.go:117] "RemoveContainer" containerID="2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806" Jul 16 00:02:57.078751 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:02:57.078732 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806\": container with ID starting with 2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806 not found: ID does not exist" containerID="2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806" Jul 16 00:02:57.078791 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.078759 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806"} err="failed to get container status \"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806\": rpc error: code = NotFound desc = could not find container \"2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806\": container with ID starting with 2c1cdb5c9b64f976d26fce94c6c32f6bf46f530f1661b0a9cca4a6e2f6033806 not found: ID does not exist" Jul 16 00:02:57.079036 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.079006 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:02:57.084652 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.084621 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2a1132-openshift-default-84df7cfd5f-dcxtt"] Jul 16 00:02:57.302989 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:57.302953 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" path="/var/lib/kubelet/pods/d0534ffd-4c8b-4992-861e-29f84a819896/volumes" Jul 16 00:02:59.286595 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:02:59.286558 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:03:00.262879 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.262843 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:00.263626 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.263602 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.263771 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.263759 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.263864 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.263854 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.263939 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.263930 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.264164 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.264149 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac45dc55-a736-4122-a421-922232cb9ecc" containerName="istio-proxy" Jul 16 00:03:00.264258 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.264248 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.264332 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.264323 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0534ffd-4c8b-4992-861e-29f84a819896" containerName="istio-proxy" Jul 16 00:03:00.277151 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.275474 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.287151 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.282600 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:00.287151 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.282852 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cb6fd1-openshift-default-dockercfg-b2p5c\"" Jul 16 00:03:00.401698 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401659 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401698 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401701 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401733 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401814 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401869 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401890 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.401962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401953 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.402160 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.401997 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqz49\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.402160 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.402022 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.502967 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.502908 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.502987 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503020 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503061 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503102 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqz49\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503167 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503200 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503554 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503230 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503554 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503274 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503669 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503644 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503725 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503692 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.503882 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503845 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.504006 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503886 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.504006 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.503943 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.505759 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.505732 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.505947 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.505930 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.512330 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.512296 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.512462 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.512426 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqz49\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49\") pod \"e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.603329 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.603227 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:00.788510 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.787601 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:00.792039 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:03:00.792002 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95824eef_57e6_40f0_843a_14ce0a988918.slice/crio-ed3be6a9b52bfd529a1ec391364b28771a69e957d72d475cb4588411d7788256 WatchSource:0}: Error finding container ed3be6a9b52bfd529a1ec391364b28771a69e957d72d475cb4588411d7788256: Status 404 returned error can't find the container with id ed3be6a9b52bfd529a1ec391364b28771a69e957d72d475cb4588411d7788256 Jul 16 00:03:00.794326 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.794282 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:00.794414 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.794387 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:00.794458 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:00.794431 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:01.082005 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:01.081962 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerStarted","Data":"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db"} Jul 16 00:03:01.082227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:01.082035 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerStarted","Data":"ed3be6a9b52bfd529a1ec391364b28771a69e957d72d475cb4588411d7788256"} Jul 16 00:03:01.102712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:01.102654 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podStartSLOduration=1.102635871 podStartE2EDuration="1.102635871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:03:01.101908485 +0000 UTC m=+1970.390723914" watchObservedRunningTime="2026-07-16 00:03:01.102635871 +0000 UTC m=+1970.391451300" Jul 16 00:03:01.604051 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:01.604008 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:01.609448 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:01.609415 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:02.091604 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:02.091527 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:02.092791 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:02.092767 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:08.130642 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.130607 2551 generic.go:356] "Generic (PLEG): container finished" podID="7069e248-241a-4eba-b5d9-a89601edfffd" containerID="6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd" exitCode=0 Jul 16 00:03:08.131097 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.130682 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerDied","Data":"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd"} Jul 16 00:03:08.131190 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.131096 2551 scope.go:117] "RemoveContainer" containerID="6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd" Jul 16 00:03:08.132147 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.132102 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:08.132214 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.132186 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:08.132309 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:08.132218 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:09.038681 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:09.038627 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:09.137976 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:09.137931 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerStarted","Data":"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e"} Jul 16 00:03:09.178185 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:09.178119 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:09.178404 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:09.178330 2551 patch_prober.go:28] interesting pod/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 00:03:09.178491 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:09.178396 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 00:03:10.177296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:10.177256 2551 patch_prober.go:28] interesting pod/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 00:03:10.178304 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:10.177866 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 00:03:11.178176 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:11.177624 2551 patch_prober.go:28] interesting pod/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 00:03:11.178176 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:11.177705 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 00:03:11.794418 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:11.794381 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:03:12.177620 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:12.177519 2551 patch_prober.go:28] interesting pod/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 16 00:03:12.177907 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:12.177876 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 16 00:03:13.189348 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:13.189314 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:13.189843 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:13.189611 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:13.190802 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:13.190779 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:21.217763 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.217728 2551 generic.go:356] "Generic (PLEG): container finished" podID="95824eef-57e6-40f0-843a-14ce0a988918" containerID="240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db" exitCode=0 Jul 16 00:03:21.218333 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.217814 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerDied","Data":"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db"} Jul 16 00:03:21.218408 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.218392 2551 scope.go:117] "RemoveContainer" containerID="240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db" Jul 16 00:03:21.219610 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.219578 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:21.219697 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.219681 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:21.219738 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:21.219728 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:22.225062 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:22.225030 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerStarted","Data":"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d"} Jul 16 00:03:22.604426 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:22.604384 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:22.604653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:22.604631 2551 patch_prober.go:28] interesting pod/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 00:03:22.604706 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:22.604686 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 00:03:23.604554 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:23.604512 2551 patch_prober.go:28] interesting pod/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 00:03:23.604947 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:23.604573 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 00:03:24.603870 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:24.603827 2551 patch_prober.go:28] interesting pod/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 00:03:24.604070 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:24.603890 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 00:03:25.604008 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:25.603964 2551 patch_prober.go:28] interesting pod/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 16 00:03:25.604557 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:25.604043 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 16 00:03:26.607838 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:26.607805 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:26.608248 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:26.608218 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:26.608962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:26.608942 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:26.790259 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:26.790214 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:03:27.572414 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:27.572377 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:03:27.572670 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:27.572638 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" containerID="cri-o://502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e" gracePeriod=30 Jul 16 00:03:28.192237 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:28.192188 2551 patch_prober.go:28] interesting pod/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:03:28.192661 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:28.192263 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:03:32.822181 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.822152 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:32.912615 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912502 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912615 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912554 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912615 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912590 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912626 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912653 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d2jp\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912806 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.912905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912886 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.913166 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912926 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.913166 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912962 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert\") pod \"7069e248-241a-4eba-b5d9-a89601edfffd\" (UID: \"7069e248-241a-4eba-b5d9-a89601edfffd\") " Jul 16 00:03:32.913166 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.912976 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:32.913166 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913111 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:32.913380 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913199 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:32.913438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913373 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:32.913438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913395 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:32.913438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913410 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:32.913594 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913441 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:03:32.913594 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.913498 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data" (OuterVolumeSpecName: "istio-data") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:32.914943 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.914908 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp" (OuterVolumeSpecName: "kube-api-access-6d2jp") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "kube-api-access-6d2jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:32.914943 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.914906 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:32.915157 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.914997 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token" (OuterVolumeSpecName: "istio-token") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:32.915157 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:32.915120 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7069e248-241a-4eba-b5d9-a89601edfffd" (UID: "7069e248-241a-4eba-b5d9-a89601edfffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:03:33.014535 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014494 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7069e248-241a-4eba-b5d9-a89601edfffd-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.014535 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014535 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7069e248-241a-4eba-b5d9-a89601edfffd-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.014752 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014551 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.014752 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014565 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.014752 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014577 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7069e248-241a-4eba-b5d9-a89601edfffd-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.014752 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.014590 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2jp\" (UniqueName: \"kubernetes.io/projected/7069e248-241a-4eba-b5d9-a89601edfffd-kube-api-access-6d2jp\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:33.283391 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.283352 2551 generic.go:356] "Generic (PLEG): container finished" podID="7069e248-241a-4eba-b5d9-a89601edfffd" containerID="502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e" exitCode=0 Jul 16 00:03:33.283391 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.283398 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerDied","Data":"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e"} Jul 16 00:03:33.283669 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.283421 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" Jul 16 00:03:33.283669 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.283481 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8" event={"ID":"7069e248-241a-4eba-b5d9-a89601edfffd","Type":"ContainerDied","Data":"ebe058abaa7c78a1737f8b698c349ca1a34c882064e203b5b7a08fc133496af5"} Jul 16 00:03:33.283669 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.283514 2551 scope.go:117] "RemoveContainer" containerID="502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e" Jul 16 00:03:33.293252 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.293227 2551 scope.go:117] "RemoveContainer" containerID="6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd" Jul 16 00:03:33.302501 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.302454 2551 scope.go:117] "RemoveContainer" containerID="502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e" Jul 16 00:03:33.302820 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:03:33.302793 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e\": container with ID starting with 502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e not found: ID does not exist" containerID="502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e" Jul 16 00:03:33.303009 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.302867 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e"} err="failed to get container status \"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e\": rpc error: code = NotFound desc = could not find container \"502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e\": container with ID starting with 502c72ebe0188532a2a226e7da6ad6823ae645cd97cbaf60cd9a1d5f0407069e not found: ID does not exist" Jul 16 00:03:33.303009 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.302946 2551 scope.go:117] "RemoveContainer" containerID="6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd" Jul 16 00:03:33.303302 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:03:33.303270 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd\": container with ID starting with 6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd not found: ID does not exist" containerID="6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd" Jul 16 00:03:33.303360 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.303310 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd"} err="failed to get container status \"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd\": rpc error: code = NotFound desc = could not find container \"6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd\": container with ID starting with 6bb5e13cf4d3cf1f16fcd7469eaf5264264d3658ee8f8d6684a9c343ee85d8fd not found: ID does not exist" Jul 16 00:03:33.306472 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.306437 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:03:33.309426 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:33.309392 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3cb745-openshift-default-fd5b5fbb7-4xjm8"] Jul 16 00:03:35.303360 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:35.303328 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" path="/var/lib/kubelet/pods/7069e248-241a-4eba-b5d9-a89601edfffd/volumes" Jul 16 00:03:35.884805 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:35.884763 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:03:40.096747 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.096707 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:40.097346 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.097005 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" containerID="cri-o://bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d" gracePeriod=30 Jul 16 00:03:40.979507 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.979467 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:03:40.979976 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.979956 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.979976 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.979976 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.980110 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.979989 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.980110 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.979995 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.980110 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.980060 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.980110 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.980070 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7069e248-241a-4eba-b5d9-a89601edfffd" containerName="istio-proxy" Jul 16 00:03:40.983281 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.983255 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:40.985949 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.985925 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-079998-openshift-default-dockercfg-55722\"" Jul 16 00:03:40.994218 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:40.994186 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:03:41.083319 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083274 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083319 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083314 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083354 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083440 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083480 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083717 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083580 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083717 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083630 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zkf\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083795 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083714 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.083795 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.083737 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.184722 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184674 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zkf\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184766 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184790 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184826 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184855 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184905 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184937 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184963 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.184999 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.185260 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.185298 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.185362 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.185577 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.185712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.185689 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.187659 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.187624 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.187830 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.187810 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.193871 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.193835 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.194077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.194047 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zkf\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf\") pod \"e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.295622 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.295588 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:41.452193 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.452159 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:03:41.454847 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:03:41.454817 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3630454c_eff4_460d_9213_7c9cdc0c0881.slice/crio-349d3083f562e0a3f16f181559dd824320ba035aca07d7748a31c63727415a00 WatchSource:0}: Error finding container 349d3083f562e0a3f16f181559dd824320ba035aca07d7748a31c63727415a00: Status 404 returned error can't find the container with id 349d3083f562e0a3f16f181559dd824320ba035aca07d7748a31c63727415a00 Jul 16 00:03:41.457684 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.457647 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:41.457775 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.457726 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:41.457775 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.457757 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:41.611088 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.610968 2551 patch_prober.go:28] interesting pod/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:03:41.611088 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:41.611044 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:03:42.333312 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:42.333275 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerStarted","Data":"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564"} Jul 16 00:03:42.333809 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:42.333335 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerStarted","Data":"349d3083f562e0a3f16f181559dd824320ba035aca07d7748a31c63727415a00"} Jul 16 00:03:42.353511 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:42.353455 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podStartSLOduration=2.35343753 podStartE2EDuration="2.35343753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:03:42.351805197 +0000 UTC m=+2011.640620625" watchObservedRunningTime="2026-07-16 00:03:42.35343753 +0000 UTC m=+2011.642252958" Jul 16 00:03:43.303812 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:43.303781 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:43.303996 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:43.303878 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:43.338409 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:43.338377 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:43.339463 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:43.339443 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:03:45.348052 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.348023 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:45.350104 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.350070 2551 generic.go:356] "Generic (PLEG): container finished" podID="95824eef-57e6-40f0-843a-14ce0a988918" containerID="bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d" exitCode=0 Jul 16 00:03:45.350270 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.350118 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" Jul 16 00:03:45.350270 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.350154 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerDied","Data":"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d"} Jul 16 00:03:45.350270 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.350217 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb" event={"ID":"95824eef-57e6-40f0-843a-14ce0a988918","Type":"ContainerDied","Data":"ed3be6a9b52bfd529a1ec391364b28771a69e957d72d475cb4588411d7788256"} Jul 16 00:03:45.350270 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.350249 2551 scope.go:117] "RemoveContainer" containerID="bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d" Jul 16 00:03:45.359242 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.359219 2551 scope.go:117] "RemoveContainer" containerID="240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db" Jul 16 00:03:45.369283 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.369256 2551 scope.go:117] "RemoveContainer" containerID="bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d" Jul 16 00:03:45.369816 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:03:45.369760 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d\": container with ID starting with bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d not found: ID does not exist" containerID="bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d" Jul 16 00:03:45.369816 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.369805 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d"} err="failed to get container status \"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d\": rpc error: code = NotFound desc = could not find container \"bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d\": container with ID starting with bd80baebdb01e8a0641613d68c4758147f80888ccc167cb93b99ac3048ea2f3d not found: ID does not exist" Jul 16 00:03:45.370018 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.369831 2551 scope.go:117] "RemoveContainer" containerID="240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db" Jul 16 00:03:45.370198 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:03:45.370176 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db\": container with ID starting with 240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db not found: ID does not exist" containerID="240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db" Jul 16 00:03:45.370266 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.370207 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db"} err="failed to get container status \"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db\": rpc error: code = NotFound desc = could not find container \"240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db\": container with ID starting with 240d401895175381b1a5d1e3eebb7f2c7ad01935ffff791c0ebea04b542a67db not found: ID does not exist" Jul 16 00:03:45.423520 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423478 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423520 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423525 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423554 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423577 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423608 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423636 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqz49\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423674 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423701 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.423783 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423759 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs\") pod \"95824eef-57e6-40f0-843a-14ce0a988918\" (UID: \"95824eef-57e6-40f0-843a-14ce0a988918\") " Jul 16 00:03:45.424162 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.423879 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:45.424162 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.424112 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:03:45.424340 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.424308 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:45.424943 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.424914 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:45.425064 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.425032 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data" (OuterVolumeSpecName: "istio-data") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:45.425344 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.425213 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.426936 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.426749 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:03:45.427309 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.427271 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:03:45.427309 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.427280 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49" (OuterVolumeSpecName: "kube-api-access-dqz49") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "kube-api-access-dqz49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:45.427552 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.427520 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token" (OuterVolumeSpecName: "istio-token") pod "95824eef-57e6-40f0-843a-14ce0a988918" (UID: "95824eef-57e6-40f0-843a-14ce0a988918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:03:45.526296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526253 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526286 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526295 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526303 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95824eef-57e6-40f0-843a-14ce0a988918-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526311 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95824eef-57e6-40f0-843a-14ce0a988918-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526614 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526319 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95824eef-57e6-40f0-843a-14ce0a988918-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526614 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526327 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.526614 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.526335 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqz49\" (UniqueName: \"kubernetes.io/projected/95824eef-57e6-40f0-843a-14ce0a988918-kube-api-access-dqz49\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:03:45.675828 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.675789 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:45.679748 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:45.679715 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cb6fd1-openshift-default-5db644787c-zkxcb"] Jul 16 00:03:47.302916 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:47.302875 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95824eef-57e6-40f0-843a-14ce0a988918" path="/var/lib/kubelet/pods/95824eef-57e6-40f0-843a-14ce0a988918/volumes" Jul 16 00:03:49.359971 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.359934 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:03:49.360398 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.360329 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:03:49.360398 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.360342 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:03:49.360398 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.360354 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:03:49.360398 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.360360 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:03:49.360533 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.360423 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:03:49.363430 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.363401 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.365959 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.365935 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-91d115-openshift-default-dockercfg-sznxg\"" Jul 16 00:03:49.377755 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.377718 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:03:49.415001 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.414943 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:03:49.458620 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458582 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.458859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458648 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.458859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458669 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.458859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458738 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.458859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458798 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7vk\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.458859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458825 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.459149 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458868 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.459149 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458915 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.459149 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.458958 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560199 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560258 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560300 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560330 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560348 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560422 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560470 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7vk\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560540 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560516 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560549 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560705 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.560850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560755 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.561013 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560912 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.561013 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.560987 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.561179 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.561153 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.563197 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.563165 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.563350 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.563212 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.568777 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.568747 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.568923 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.568901 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7vk\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk\") pod \"e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.676562 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.676470 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:49.832923 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.832893 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:03:49.835674 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:03:49.835627 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c29d621_cad1_46d8_b2bc_6f1458b78705.slice/crio-188a9dc97733261324994effb0d564f670306015ffcfd9732764a82ea13bcf09 WatchSource:0}: Error finding container 188a9dc97733261324994effb0d564f670306015ffcfd9732764a82ea13bcf09: Status 404 returned error can't find the container with id 188a9dc97733261324994effb0d564f670306015ffcfd9732764a82ea13bcf09 Jul 16 00:03:49.838201 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.838160 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:49.838319 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.838243 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:49.838319 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:49.838274 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:50.383278 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.383173 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerStarted","Data":"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6"} Jul 16 00:03:50.383278 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.383240 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerStarted","Data":"188a9dc97733261324994effb0d564f670306015ffcfd9732764a82ea13bcf09"} Jul 16 00:03:50.410075 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.410005 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podStartSLOduration=1.409983578 podStartE2EDuration="1.409983578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:03:50.409451701 +0000 UTC m=+2019.698267130" watchObservedRunningTime="2026-07-16 00:03:50.409983578 +0000 UTC m=+2019.698799008" Jul 16 00:03:50.678020 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.677899 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:50.679639 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.679595 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:03:50.679813 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:50.679667 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:03:51.695829 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:51.695792 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:52.396588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:52.396558 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:52.397788 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:52.397760 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:03:58.340624 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.340531 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 16 00:03:58.340624 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.340607 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 16 00:03:58.430138 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.430100 2551 generic.go:356] "Generic (PLEG): container finished" podID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerID="dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564" exitCode=0 Jul 16 00:03:58.430329 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.430167 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerDied","Data":"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564"} Jul 16 00:03:58.430656 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.430640 2551 scope.go:117] "RemoveContainer" containerID="dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564" Jul 16 00:03:58.431796 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.431757 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:58.431903 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.431856 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:58.431903 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:58.431897 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:03:59.438567 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:03:59.438530 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerStarted","Data":"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a"} Jul 16 00:04:00.296114 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:00.296060 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:04:00.296427 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:00.296405 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 16 00:04:00.296518 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:00.296450 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 16 00:04:01.295988 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:01.295950 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 16 00:04:01.296398 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:01.296012 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 16 00:04:01.394378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:01.394339 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:04:02.296674 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:02.296627 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 16 00:04:02.297083 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:02.296693 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 16 00:04:03.296315 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.296272 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 16 00:04:03.296586 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.296371 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 16 00:04:03.466378 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.466257 2551 generic.go:356] "Generic (PLEG): container finished" podID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerID="7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6" exitCode=0 Jul 16 00:04:03.466887 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.466411 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerDied","Data":"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6"} Jul 16 00:04:03.466887 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.466880 2551 scope.go:117] "RemoveContainer" containerID="7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6" Jul 16 00:04:03.468562 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.468511 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:04:03.468670 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.468623 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:04:03.468735 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:03.468675 2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 16 00:04:04.301718 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.301682 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:04:04.302058 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.302041 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:04:04.302933 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.302914 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:04:04.474888 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.474850 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerStarted","Data":"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795"} Jul 16 00:04:04.679016 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.678842 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:04:04.679437 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.679406 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:04:04.679634 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:04.679601 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:04:05.547251 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.547200 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:04:05.547675 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.547654 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="95824eef-57e6-40f0-843a-14ce0a988918" containerName="istio-proxy" Jul 16 00:04:05.549807 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.549785 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.553533 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.553421 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"openshift-service-ca.crt\"" Jul 16 00:04:05.553533 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.553450 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"auth-test-model-079998-kserve-self-signed-certs\"" Jul 16 00:04:05.553533 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.553494 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"default-dockercfg-lhfpn\"" Jul 16 00:04:05.553805 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.553500 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"kube-root-ca.crt\"" Jul 16 00:04:05.561077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.561051 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:04:05.627498 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627460 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.627708 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627526 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.627708 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627640 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.627708 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627690 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.627905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627804 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.627968 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.627929 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.679152 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.677613 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:04:05.679152 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.677697 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:04:05.728567 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728524 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.728567 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728566 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.728821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728596 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.728821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728637 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.728821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728687 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.728821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.728732 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.729037 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.729016 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.729096 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.729058 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.729203 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.729180 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.731039 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.731018 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.731322 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.731304 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.737072 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.737036 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr\") pod \"auth-test-model-079998-kserve-7b68b987f-5xrws\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:05.861770 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:05.861667 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:06.053247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.053216 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:04:06.055247 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:04:06.055214 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec6b28cc_70e4_4bf2_ad68_b20f363c707b.slice/crio-2d50d338d773dee953ed557138d2742434c85d43d3ba52d3e76437ef61fb4ad8 WatchSource:0}: Error finding container 2d50d338d773dee953ed557138d2742434c85d43d3ba52d3e76437ef61fb4ad8: Status 404 returned error can't find the container with id 2d50d338d773dee953ed557138d2742434c85d43d3ba52d3e76437ef61fb4ad8 Jul 16 00:04:06.492096 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.491993 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" event={"ID":"ec6b28cc-70e4-4bf2-ad68-b20f363c707b","Type":"ContainerStarted","Data":"87101edc11f438d4c67ac4022bb46a111571d0b51eed65743ef77e99f6a543f6"} Jul 16 00:04:06.492096 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.492043 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" event={"ID":"ec6b28cc-70e4-4bf2-ad68-b20f363c707b","Type":"ContainerStarted","Data":"2d50d338d773dee953ed557138d2742434c85d43d3ba52d3e76437ef61fb4ad8"} Jul 16 00:04:06.510149 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.510068 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" podStartSLOduration=1.510047986 podStartE2EDuration="1.510047986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:04:06.508628046 +0000 UTC m=+2035.797443475" watchObservedRunningTime="2026-07-16 00:04:06.510047986 +0000 UTC m=+2035.798863417" Jul 16 00:04:06.677296 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.677246 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:04:06.677819 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:06.677324 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:04:07.397213 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:07.397160 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:04:07.677673 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:07.677570 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:04:07.678288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:07.678251 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:04:08.677597 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:08.677549 2551 patch_prober.go:28] interesting pod/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 16 00:04:08.677849 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:08.677624 2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 16 00:04:09.681604 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:09.681570 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:04:09.682692 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:09.682670 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:04:11.587010 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:11.586958 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:04:15.862117 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:15.862072 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:15.862661 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:15.862203 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:15.875467 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:15.875433 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:16.556392 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:16.556356 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:04:42.586759 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:42.586721 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:04:53.594062 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:04:53.594019 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:05:11.386312 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:05:11.386270 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:05:28.387954 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:05:28.387911 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:05:47.290379 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:05:47.290330 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:06:05.493208 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:05.493101 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:06:19.692345 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:19.692303 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:06:39.387097 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:39.387052 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:06:43.171097 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:43.171058 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:06:55.494638 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:55.494600 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:06:58.192043 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.192007 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" podUID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" containerName="main" containerID="cri-o://87101edc11f438d4c67ac4022bb46a111571d0b51eed65743ef77e99f6a543f6" gracePeriod=45 Jul 16 00:06:58.398374 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.398341 2551 generic.go:356] "Generic (PLEG): container finished" podID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" containerID="87101edc11f438d4c67ac4022bb46a111571d0b51eed65743ef77e99f6a543f6" exitCode=0 Jul 16 00:06:58.398546 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.398409 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" event={"ID":"ec6b28cc-70e4-4bf2-ad68-b20f363c707b","Type":"ContainerDied","Data":"87101edc11f438d4c67ac4022bb46a111571d0b51eed65743ef77e99f6a543f6"} Jul 16 00:06:58.443742 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.443662 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:06:58.532837 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532799 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532861 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532909 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532945 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532961 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.532994 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir\") pod \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\" (UID: \"ec6b28cc-70e4-4bf2-ad68-b20f363c707b\") " Jul 16 00:06:58.533672 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.533295 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home" (OuterVolumeSpecName: "home") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:58.534004 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.533276 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:58.534151 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.534011 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:58.536305 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.536267 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr" (OuterVolumeSpecName: "kube-api-access-vprrr") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "kube-api-access-vprrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:06:58.536505 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.536466 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm" (OuterVolumeSpecName: "dshm") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:06:58.542087 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.542047 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec6b28cc-70e4-4bf2-ad68-b20f363c707b" (UID: "ec6b28cc-70e4-4bf2-ad68-b20f363c707b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:06:58.634698 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634655 2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-dshm\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.634698 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634700 2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-model-cache\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.634922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634714 2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-home\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.634922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634759 2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tmp-dir\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.634922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634776 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprrr\" (UniqueName: \"kubernetes.io/projected/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-kube-api-access-vprrr\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.634922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.634791 2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec6b28cc-70e4-4bf2-ad68-b20f363c707b-tls-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:06:58.651953 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.651915 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:06:58.652359 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.652342 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" containerName="main" Jul 16 00:06:58.652359 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.652359 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" containerName="main" Jul 16 00:06:58.652530 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.652435 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" containerName="main" Jul 16 00:06:58.654613 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.654588 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.657469 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.657435 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"sub-test-model-079998-kserve-self-signed-certs\"" Jul 16 00:06:58.678969 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.678929 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:06:58.735497 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735401 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d755w\" (UniqueName: \"kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.735497 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735454 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.735497 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735476 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.735724 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735558 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.735724 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735627 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.735724 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.735651 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.836798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836749 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.836798 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836800 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837075 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836840 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d755w\" (UniqueName: \"kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837075 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836878 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837075 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836902 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837075 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.836958 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837326 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.837280 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837396 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.837336 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.837482 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.837461 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.839360 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.839339 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.839549 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.839528 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.854056 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.854023 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d755w\" (UniqueName: \"kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w\") pod \"sub-test-model-079998-kserve-8456895659-f2mml\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:58.966079 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:58.966033 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:06:59.114197 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.114163 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:06:59.116962 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:06:59.116931 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc57fa356_8ec8_4d68_8cf1_26ff4512ad38.slice/crio-9b3b13b751169302a6610e841817aa313d4de89867e93d86a87d38292110a191 WatchSource:0}: Error finding container 9b3b13b751169302a6610e841817aa313d4de89867e93d86a87d38292110a191: Status 404 returned error can't find the container with id 9b3b13b751169302a6610e841817aa313d4de89867e93d86a87d38292110a191 Jul 16 00:06:59.405754 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.405718 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" event={"ID":"c57fa356-8ec8-4d68-8cf1-26ff4512ad38","Type":"ContainerStarted","Data":"5da7740390a830a28d3ebacb706a90935867497a8cededa917e6627314de3ec0"} Jul 16 00:06:59.406236 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.405768 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" event={"ID":"c57fa356-8ec8-4d68-8cf1-26ff4512ad38","Type":"ContainerStarted","Data":"9b3b13b751169302a6610e841817aa313d4de89867e93d86a87d38292110a191"} Jul 16 00:06:59.407276 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.407247 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" event={"ID":"ec6b28cc-70e4-4bf2-ad68-b20f363c707b","Type":"ContainerDied","Data":"2d50d338d773dee953ed557138d2742434c85d43d3ba52d3e76437ef61fb4ad8"} Jul 16 00:06:59.407416 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.407296 2551 scope.go:117] "RemoveContainer" containerID="87101edc11f438d4c67ac4022bb46a111571d0b51eed65743ef77e99f6a543f6" Jul 16 00:06:59.407416 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.407259 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws" Jul 16 00:06:59.428714 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.428650 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" podStartSLOduration=1.428628026 podStartE2EDuration="1.428628026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:06:59.426887344 +0000 UTC m=+2208.715702770" watchObservedRunningTime="2026-07-16 00:06:59.428628026 +0000 UTC m=+2208.717443532" Jul 16 00:06:59.443799 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.443757 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:06:59.448206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:06:59.448169 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-079998/auth-test-model-079998-kserve-7b68b987f-5xrws"] Jul 16 00:07:01.303923 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:01.303886 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6b28cc-70e4-4bf2-ad68-b20f363c707b" path="/var/lib/kubelet/pods/ec6b28cc-70e4-4bf2-ad68-b20f363c707b/volumes" Jul 16 00:07:04.387692 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:04.387643 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:07:08.966480 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:08.966442 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:07:08.966894 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:08.966504 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:07:08.979414 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:08.979376 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:07:09.470288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:09.470257 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:07:38.793776 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:38.793732 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:07:53.195009 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:07:53.194964 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:08:07.618247 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:07.618209 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:08:12.187814 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:12.187774 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:08:17.228521 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.228485 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:08:17.230805 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.230782 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.233769 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.233744 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-079998\"/\"rate-test-model-079998-kserve-self-signed-certs\"" Jul 16 00:08:17.239744 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.239716 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:08:17.277920 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.277880 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qsxl\" (UniqueName: \"kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.278112 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.277947 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.278112 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.278018 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.278112 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.278065 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.278112 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.278082 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.278295 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.278213 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.379730 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.379684 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.379953 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.379763 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.379953 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.379803 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.379953 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.379823 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.380156 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.379981 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.380156 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.380057 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qsxl\" (UniqueName: \"kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.380298 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.380272 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.380413 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.380397 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.380486 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.380419 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.382810 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.382781 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.382956 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.382939 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.388482 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.388450 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qsxl\" (UniqueName: \"kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl\") pod \"rate-test-model-079998-kserve-75bd66bff5-n5d8m\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.540986 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.540948 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:17.686514 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.686484 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:08:17.688609 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:08:17.688574 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd70d526_84bf_48cc_94d3_2d4c64ad41c2.slice/crio-30e1923efc9c3c7001c7975769df9607ccf29799940b64b8d8152d134959dcc6 WatchSource:0}: Error finding container 30e1923efc9c3c7001c7975769df9607ccf29799940b64b8d8152d134959dcc6: Status 404 returned error can't find the container with id 30e1923efc9c3c7001c7975769df9607ccf29799940b64b8d8152d134959dcc6 Jul 16 00:08:17.824333 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.824165 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" event={"ID":"fd70d526-84bf-48cc-94d3-2d4c64ad41c2","Type":"ContainerStarted","Data":"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224"} Jul 16 00:08:17.824333 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.824227 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" event={"ID":"fd70d526-84bf-48cc-94d3-2d4c64ad41c2","Type":"ContainerStarted","Data":"30e1923efc9c3c7001c7975769df9607ccf29799940b64b8d8152d134959dcc6"} Jul 16 00:08:17.844049 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:17.843774 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" podStartSLOduration=0.84374652 podStartE2EDuration="843.74652ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:08:17.841052111 +0000 UTC m=+2287.129867538" watchObservedRunningTime="2026-07-16 00:08:17.84374652 +0000 UTC m=+2287.132561950" Jul 16 00:08:22.640423 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:22.640357 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" podUID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" containerName="main" containerID="cri-o://5da7740390a830a28d3ebacb706a90935867497a8cededa917e6627314de3ec0" gracePeriod=45 Jul 16 00:08:22.856100 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:22.856052 2551 generic.go:356] "Generic (PLEG): container finished" podID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" containerID="5da7740390a830a28d3ebacb706a90935867497a8cededa917e6627314de3ec0" exitCode=0 Jul 16 00:08:22.856100 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:22.856085 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" event={"ID":"c57fa356-8ec8-4d68-8cf1-26ff4512ad38","Type":"ContainerDied","Data":"5da7740390a830a28d3ebacb706a90935867497a8cededa917e6627314de3ec0"} Jul 16 00:08:22.906491 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:22.906394 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:08:23.037323 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037283 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037357 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d755w\" (UniqueName: \"kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037378 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037415 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037445 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037509 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037495 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache\") pod \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\" (UID: \"c57fa356-8ec8-4d68-8cf1-26ff4512ad38\") " Jul 16 00:08:23.037778 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037599 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home" (OuterVolumeSpecName: "home") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:08:23.037778 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037664 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:08:23.037778 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037770 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache" (OuterVolumeSpecName: "model-cache") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:08:23.037964 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037835 2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-home\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.037964 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037861 2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tmp-dir\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.037964 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.037879 2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-model-cache\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.039605 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.039568 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm" (OuterVolumeSpecName: "dshm") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:08:23.040044 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.040014 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w" (OuterVolumeSpecName: "kube-api-access-d755w") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "kube-api-access-d755w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:08:23.040194 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.040063 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c57fa356-8ec8-4d68-8cf1-26ff4512ad38" (UID: "c57fa356-8ec8-4d68-8cf1-26ff4512ad38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:08:23.138764 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.138722 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d755w\" (UniqueName: \"kubernetes.io/projected/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-kube-api-access-d755w\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.138764 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.138765 2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-dshm\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.138995 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.138781 2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57fa356-8ec8-4d68-8cf1-26ff4512ad38-tls-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:08:23.863045 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.863000 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" event={"ID":"c57fa356-8ec8-4d68-8cf1-26ff4512ad38","Type":"ContainerDied","Data":"9b3b13b751169302a6610e841817aa313d4de89867e93d86a87d38292110a191"} Jul 16 00:08:23.863536 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.863052 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml" Jul 16 00:08:23.863536 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.863064 2551 scope.go:117] "RemoveContainer" containerID="5da7740390a830a28d3ebacb706a90935867497a8cededa917e6627314de3ec0" Jul 16 00:08:23.884273 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.884234 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:08:23.890896 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:23.890861 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-079998/sub-test-model-079998-kserve-8456895659-f2mml"] Jul 16 00:08:25.303343 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:25.303304 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" path="/var/lib/kubelet/pods/c57fa356-8ec8-4d68-8cf1-26ff4512ad38/volumes" Jul 16 00:08:27.541859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:27.541797 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:27.541859 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:27.541876 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:27.554726 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:27.554697 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:27.895832 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:27.895732 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:08:28.381281 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:28.381244 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:08:49.795286 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:08:49.795242 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:09:06.393046 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:06.392995 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:09:25.486821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:25.486782 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:09:29.757820 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:29.757777 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:09:31.603773 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.603728 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644608174640"] Jul 16 00:09:31.604256 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.604090 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" containerName="main" Jul 16 00:09:31.604256 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.604103 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" containerName="main" Jul 16 00:09:31.604256 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.604189 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57fa356-8ec8-4d68-8cf1-26ff4512ad38" containerName="main" Jul 16 00:09:31.606858 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.606836 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:31.609744 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.609716 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9k447\"" Jul 16 00:09:31.612428 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.612399 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644608174640"] Jul 16 00:09:31.670364 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.670331 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69djx\" (UniqueName: \"kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx\") pod \"test-curl-7487-139644608174640\" (UID: \"e6351669-5ccc-46fd-8922-48ff72500817\") " pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:31.770805 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.770772 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69djx\" (UniqueName: \"kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx\") pod \"test-curl-7487-139644608174640\" (UID: \"e6351669-5ccc-46fd-8922-48ff72500817\") " pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:31.778876 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.778839 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69djx\" (UniqueName: \"kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx\") pod \"test-curl-7487-139644608174640\" (UID: \"e6351669-5ccc-46fd-8922-48ff72500817\") " pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:31.917905 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:31.917812 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:32.060641 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:32.060611 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644608174640"] Jul 16 00:09:32.062528 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:32.062499 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6351669_5ccc_46fd_8922_48ff72500817.slice/crio-ca17637c011013ca837be525ed5aa096636708949f34ac1d119385f01209bd81 WatchSource:0}: Error finding container ca17637c011013ca837be525ed5aa096636708949f34ac1d119385f01209bd81: Status 404 returned error can't find the container with id ca17637c011013ca837be525ed5aa096636708949f34ac1d119385f01209bd81 Jul 16 00:09:32.063776 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:32.063755 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 00:09:32.228014 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:32.227911 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644608174640" event={"ID":"e6351669-5ccc-46fd-8922-48ff72500817","Type":"ContainerStarted","Data":"ca17637c011013ca837be525ed5aa096636708949f34ac1d119385f01209bd81"} Jul 16 00:09:33.234973 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:33.234934 2551 generic.go:356] "Generic (PLEG): container finished" podID="e6351669-5ccc-46fd-8922-48ff72500817" containerID="cefcf629ec0a8241f97e17176e6881a558dfc3154d3a2f97ebabcbc04092f73d" exitCode=0 Jul 16 00:09:33.235432 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:33.235025 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644608174640" event={"ID":"e6351669-5ccc-46fd-8922-48ff72500817","Type":"ContainerDied","Data":"cefcf629ec0a8241f97e17176e6881a558dfc3154d3a2f97ebabcbc04092f73d"} Jul 16 00:09:34.389000 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.388968 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:34.424352 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.424325 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644608174640_e6351669-5ccc-46fd-8922-48ff72500817/test-curl-7487-139644608174640/0.log" Jul 16 00:09:34.447464 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.447429 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644608174640"] Jul 16 00:09:34.451173 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.451118 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644608174640"] Jul 16 00:09:34.502481 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.502388 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69djx\" (UniqueName: \"kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx\") pod \"e6351669-5ccc-46fd-8922-48ff72500817\" (UID: \"e6351669-5ccc-46fd-8922-48ff72500817\") " Jul 16 00:09:34.504642 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.504605 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx" (OuterVolumeSpecName: "kube-api-access-69djx") pod "e6351669-5ccc-46fd-8922-48ff72500817" (UID: "e6351669-5ccc-46fd-8922-48ff72500817"). InnerVolumeSpecName "kube-api-access-69djx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:34.566121 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.566083 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644608174000"] Jul 16 00:09:34.566478 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.566463 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6351669-5ccc-46fd-8922-48ff72500817" containerName="test-curl-7487-139644608174640" Jul 16 00:09:34.566531 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.566480 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6351669-5ccc-46fd-8922-48ff72500817" containerName="test-curl-7487-139644608174640" Jul 16 00:09:34.566567 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.566544 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6351669-5ccc-46fd-8922-48ff72500817" containerName="test-curl-7487-139644608174640" Jul 16 00:09:34.569506 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.569483 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:34.578621 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.578592 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644608174000"] Jul 16 00:09:34.604018 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.603976 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvvl\" (UniqueName: \"kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl\") pod \"test-curl-7487-139644608174000\" (UID: \"ab474a45-d19d-46f2-a9e1-aefe49975e23\") " pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:34.604221 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.604172 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69djx\" (UniqueName: \"kubernetes.io/projected/e6351669-5ccc-46fd-8922-48ff72500817-kube-api-access-69djx\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:34.705478 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.705445 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvvl\" (UniqueName: \"kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl\") pod \"test-curl-7487-139644608174000\" (UID: \"ab474a45-d19d-46f2-a9e1-aefe49975e23\") " pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:34.713996 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.713965 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvvl\" (UniqueName: \"kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl\") pod \"test-curl-7487-139644608174000\" (UID: \"ab474a45-d19d-46f2-a9e1-aefe49975e23\") " pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:34.879523 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:34.879471 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:35.025791 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:35.025765 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644608174000"] Jul 16 00:09:35.027834 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:35.027806 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab474a45_d19d_46f2_a9e1_aefe49975e23.slice/crio-e27f08d7214bf8c51fedba258dd54544138b1b54568c71adb17672116eb1451f WatchSource:0}: Error finding container e27f08d7214bf8c51fedba258dd54544138b1b54568c71adb17672116eb1451f: Status 404 returned error can't find the container with id e27f08d7214bf8c51fedba258dd54544138b1b54568c71adb17672116eb1451f Jul 16 00:09:35.250625 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:35.250525 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:35.250625 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:35.250570 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca17637c011013ca837be525ed5aa096636708949f34ac1d119385f01209bd81" Jul 16 00:09:35.252037 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:35.252002 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644608174000" event={"ID":"ab474a45-d19d-46f2-a9e1-aefe49975e23","Type":"ContainerStarted","Data":"e27f08d7214bf8c51fedba258dd54544138b1b54568c71adb17672116eb1451f"} Jul 16 00:09:35.262290 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:35.262249 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644608174640\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="e6351669-5ccc-46fd-8922-48ff72500817" pod="opendatahub/test-curl-7487-139644608174640" Jul 16 00:09:35.303726 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:35.303691 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6351669-5ccc-46fd-8922-48ff72500817" path="/var/lib/kubelet/pods/e6351669-5ccc-46fd-8922-48ff72500817/volumes" Jul 16 00:09:36.260235 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:36.260199 2551 generic.go:356] "Generic (PLEG): container finished" podID="ab474a45-d19d-46f2-a9e1-aefe49975e23" containerID="863a4bc0a52cfc489a4f45366e6c81a46dc4da526e65da469cc8c6fb38101b5b" exitCode=0 Jul 16 00:09:36.260614 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:36.260287 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644608174000" event={"ID":"ab474a45-d19d-46f2-a9e1-aefe49975e23","Type":"ContainerDied","Data":"863a4bc0a52cfc489a4f45366e6c81a46dc4da526e65da469cc8c6fb38101b5b"} Jul 16 00:09:37.412418 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.412386 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:37.432266 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.432235 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644608174000_ab474a45-d19d-46f2-a9e1-aefe49975e23/test-curl-7487-139644608174000/0.log" Jul 16 00:09:37.456071 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.456029 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644608174000"] Jul 16 00:09:37.459538 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.459504 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644608174000"] Jul 16 00:09:37.532063 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.531977 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prvvl\" (UniqueName: \"kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl\") pod \"ab474a45-d19d-46f2-a9e1-aefe49975e23\" (UID: \"ab474a45-d19d-46f2-a9e1-aefe49975e23\") " Jul 16 00:09:37.534246 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.534196 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl" (OuterVolumeSpecName: "kube-api-access-prvvl") pod "ab474a45-d19d-46f2-a9e1-aefe49975e23" (UID: "ab474a45-d19d-46f2-a9e1-aefe49975e23"). InnerVolumeSpecName "kube-api-access-prvvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:37.618700 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.618658 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:37.619034 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.619019 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" containerName="test-curl-7487-139644608174000" Jul 16 00:09:37.619082 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.619035 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" containerName="test-curl-7487-139644608174000" Jul 16 00:09:37.619118 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.619111 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" containerName="test-curl-7487-139644608174000" Jul 16 00:09:37.622081 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.622058 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:37.630755 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.629668 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:37.635615 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.635582 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prvvl\" (UniqueName: \"kubernetes.io/projected/ab474a45-d19d-46f2-a9e1-aefe49975e23-kube-api-access-prvvl\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:37.736544 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.736488 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr2s\" (UniqueName: \"kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s\") pod \"test-curl-7487-139644603514144\" (UID: \"4d43836b-49d8-466d-bf2c-3b709eabf643\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:37.837463 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.837354 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr2s\" (UniqueName: \"kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s\") pod \"test-curl-7487-139644603514144\" (UID: \"4d43836b-49d8-466d-bf2c-3b709eabf643\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:37.845864 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.845831 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr2s\" (UniqueName: \"kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s\") pod \"test-curl-7487-139644603514144\" (UID: \"4d43836b-49d8-466d-bf2c-3b709eabf643\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:37.943741 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:37.943692 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:38.091612 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:38.091583 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:38.093664 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:38.093633 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d43836b_49d8_466d_bf2c_3b709eabf643.slice/crio-b238830eff5ff3fcf203926996e37dd9a0acc3070ab13c7e5e1a375325cbe951 WatchSource:0}: Error finding container b238830eff5ff3fcf203926996e37dd9a0acc3070ab13c7e5e1a375325cbe951: Status 404 returned error can't find the container with id b238830eff5ff3fcf203926996e37dd9a0acc3070ab13c7e5e1a375325cbe951 Jul 16 00:09:38.273459 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:38.273418 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644603514144" event={"ID":"4d43836b-49d8-466d-bf2c-3b709eabf643","Type":"ContainerStarted","Data":"b238830eff5ff3fcf203926996e37dd9a0acc3070ab13c7e5e1a375325cbe951"} Jul 16 00:09:38.274771 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:38.274737 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:38.274771 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:38.274764 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27f08d7214bf8c51fedba258dd54544138b1b54568c71adb17672116eb1451f" Jul 16 00:09:38.285256 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:38.285220 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644608174000\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:39.282003 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:39.281964 2551 generic.go:356] "Generic (PLEG): container finished" podID="4d43836b-49d8-466d-bf2c-3b709eabf643" containerID="28fc009d299ef0e458cfd4df7e6a9b19dee9600e8b063899c3cb0759e8d0d248" exitCode=0 Jul 16 00:09:39.282443 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:39.282049 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644603514144" event={"ID":"4d43836b-49d8-466d-bf2c-3b709eabf643","Type":"ContainerDied","Data":"28fc009d299ef0e458cfd4df7e6a9b19dee9600e8b063899c3cb0759e8d0d248"} Jul 16 00:09:39.298405 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:39.298356 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644608174000\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" pod="opendatahub/test-curl-7487-139644608174000" Jul 16 00:09:39.302694 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:39.302662 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab474a45-d19d-46f2-a9e1-aefe49975e23" path="/var/lib/kubelet/pods/ab474a45-d19d-46f2-a9e1-aefe49975e23/volumes" Jul 16 00:09:40.428544 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.428512 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:40.448661 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.448629 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644603514144_4d43836b-49d8-466d-bf2c-3b709eabf643/test-curl-7487-139644603514144/0.log" Jul 16 00:09:40.474363 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.474327 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:40.478026 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.477992 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:40.562253 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.562151 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pr2s\" (UniqueName: \"kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s\") pod \"4d43836b-49d8-466d-bf2c-3b709eabf643\" (UID: \"4d43836b-49d8-466d-bf2c-3b709eabf643\") " Jul 16 00:09:40.564337 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.564304 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s" (OuterVolumeSpecName: "kube-api-access-5pr2s") pod "4d43836b-49d8-466d-bf2c-3b709eabf643" (UID: "4d43836b-49d8-466d-bf2c-3b709eabf643"). InnerVolumeSpecName "kube-api-access-5pr2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:40.597268 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.597223 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:40.597672 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.597651 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d43836b-49d8-466d-bf2c-3b709eabf643" containerName="test-curl-7487-139644603514144" Jul 16 00:09:40.597756 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.597674 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d43836b-49d8-466d-bf2c-3b709eabf643" containerName="test-curl-7487-139644603514144" Jul 16 00:09:40.597812 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.597767 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d43836b-49d8-466d-bf2c-3b709eabf643" containerName="test-curl-7487-139644603514144" Jul 16 00:09:40.600739 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.600710 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:40.607206 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.607174 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:40.663322 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.663280 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpb9q\" (UniqueName: \"kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q\") pod \"test-curl-7487-139644603514144\" (UID: \"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:40.663492 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.663444 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr2s\" (UniqueName: \"kubernetes.io/projected/4d43836b-49d8-466d-bf2c-3b709eabf643-kube-api-access-5pr2s\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:40.764559 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.764517 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpb9q\" (UniqueName: \"kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q\") pod \"test-curl-7487-139644603514144\" (UID: \"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:40.772926 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.772873 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpb9q\" (UniqueName: \"kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q\") pod \"test-curl-7487-139644603514144\" (UID: \"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9\") " pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:40.912994 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:40.912898 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:41.057351 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.057323 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:41.059716 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:41.059690 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9ea1e2c_1ba2_49f9_95c2_0548a86064e9.slice/crio-916d6ad0a4c480b21d2d7d0f62982e7962d0da7f950d7cc952492413b0991fd0 WatchSource:0}: Error finding container 916d6ad0a4c480b21d2d7d0f62982e7962d0da7f950d7cc952492413b0991fd0: Status 404 returned error can't find the container with id 916d6ad0a4c480b21d2d7d0f62982e7962d0da7f950d7cc952492413b0991fd0 Jul 16 00:09:41.188576 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.188489 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:09:41.294686 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.294658 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:41.294686 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.294691 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b238830eff5ff3fcf203926996e37dd9a0acc3070ab13c7e5e1a375325cbe951" Jul 16 00:09:41.296023 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.295992 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644603514144" event={"ID":"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9","Type":"ContainerStarted","Data":"916d6ad0a4c480b21d2d7d0f62982e7962d0da7f950d7cc952492413b0991fd0"} Jul 16 00:09:41.303548 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:41.303516 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d43836b-49d8-466d-bf2c-3b709eabf643" path="/var/lib/kubelet/pods/4d43836b-49d8-466d-bf2c-3b709eabf643/volumes" Jul 16 00:09:42.302326 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:42.302283 2551 generic.go:356] "Generic (PLEG): container finished" podID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" containerID="22141f13ce9f1c8f7f4687cd3e8fe5dec4c969a25c12144b70a1b3d44d04dbcd" exitCode=0 Jul 16 00:09:42.302787 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:42.302380 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644603514144" event={"ID":"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9","Type":"ContainerDied","Data":"22141f13ce9f1c8f7f4687cd3e8fe5dec4c969a25c12144b70a1b3d44d04dbcd"} Jul 16 00:09:43.449433 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.449397 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:43.468534 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.468504 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644603514144_b9ea1e2c-1ba2-49f9-95c2-0548a86064e9/test-curl-7487-139644603514144/0.log" Jul 16 00:09:43.491948 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.491908 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:43.495723 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.495688 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644603514144"] Jul 16 00:09:43.592369 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.592265 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpb9q\" (UniqueName: \"kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q\") pod \"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9\" (UID: \"b9ea1e2c-1ba2-49f9-95c2-0548a86064e9\") " Jul 16 00:09:43.594439 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.594403 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q" (OuterVolumeSpecName: "kube-api-access-mpb9q") pod "b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" (UID: "b9ea1e2c-1ba2-49f9-95c2-0548a86064e9"). InnerVolumeSpecName "kube-api-access-mpb9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:43.614856 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.614807 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:43.615279 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.615260 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" containerName="test-curl-7487-139644603514144" Jul 16 00:09:43.615382 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.615281 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" containerName="test-curl-7487-139644603514144" Jul 16 00:09:43.615439 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.615415 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" containerName="test-curl-7487-139644603514144" Jul 16 00:09:43.618535 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.618505 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:43.624234 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.624200 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:43.693969 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.693932 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kr2t\" (UniqueName: \"kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t\") pod \"test-curl-7487-139644606116176\" (UID: \"6df60380-09d0-4e51-aa7b-cd905be23c87\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:43.694185 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.694005 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpb9q\" (UniqueName: \"kubernetes.io/projected/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9-kube-api-access-mpb9q\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:43.795355 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.795310 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kr2t\" (UniqueName: \"kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t\") pod \"test-curl-7487-139644606116176\" (UID: \"6df60380-09d0-4e51-aa7b-cd905be23c87\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:43.804438 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.804396 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kr2t\" (UniqueName: \"kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t\") pod \"test-curl-7487-139644606116176\" (UID: \"6df60380-09d0-4e51-aa7b-cd905be23c87\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:43.929761 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:43.929654 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:44.072292 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:44.072233 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:44.074714 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:44.074671 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df60380_09d0_4e51_aa7b_cd905be23c87.slice/crio-71520fd30aa99efbab85463b7e5caf8a4dda89563a8079d00bc83b12f7f6a4ee WatchSource:0}: Error finding container 71520fd30aa99efbab85463b7e5caf8a4dda89563a8079d00bc83b12f7f6a4ee: Status 404 returned error can't find the container with id 71520fd30aa99efbab85463b7e5caf8a4dda89563a8079d00bc83b12f7f6a4ee Jul 16 00:09:44.315287 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:44.315254 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:44.315488 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:44.315260 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916d6ad0a4c480b21d2d7d0f62982e7962d0da7f950d7cc952492413b0991fd0" Jul 16 00:09:44.316569 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:44.316541 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644606116176" event={"ID":"6df60380-09d0-4e51-aa7b-cd905be23c87","Type":"ContainerStarted","Data":"71520fd30aa99efbab85463b7e5caf8a4dda89563a8079d00bc83b12f7f6a4ee"} Jul 16 00:09:44.325883 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:44.325850 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644603514144\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:44.780214 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:44.780168 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" podUID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" containerName="main" containerID="cri-o://43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224" gracePeriod=45 Jul 16 00:09:45.040613 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.040537 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:09:45.060764 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:45.060716 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644603514144\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" pod="opendatahub/test-curl-7487-139644603514144" Jul 16 00:09:45.107067 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107018 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107306 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107077 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107306 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107146 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qsxl\" (UniqueName: \"kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107306 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107171 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107306 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107204 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107306 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107227 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache\") pod \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\" (UID: \"fd70d526-84bf-48cc-94d3-2d4c64ad41c2\") " Jul 16 00:09:45.107600 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107512 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:09:45.107600 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107522 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home" (OuterVolumeSpecName: "home") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:09:45.107600 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107567 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:09:45.107754 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107607 2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-home\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.107754 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.107624 2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tmp-dir\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.109315 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.109289 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm" (OuterVolumeSpecName: "dshm") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:09:45.109429 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.109334 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 00:09:45.109429 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.109340 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl" (OuterVolumeSpecName: "kube-api-access-9qsxl") pod "fd70d526-84bf-48cc-94d3-2d4c64ad41c2" (UID: "fd70d526-84bf-48cc-94d3-2d4c64ad41c2"). InnerVolumeSpecName "kube-api-access-9qsxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:45.208073 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.208044 2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-tls-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.208073 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.208072 2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-dshm\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.208585 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.208082 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qsxl\" (UniqueName: \"kubernetes.io/projected/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-kube-api-access-9qsxl\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.208585 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.208092 2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd70d526-84bf-48cc-94d3-2d4c64ad41c2-model-cache\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:45.303484 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.303378 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ea1e2c-1ba2-49f9-95c2-0548a86064e9" path="/var/lib/kubelet/pods/b9ea1e2c-1ba2-49f9-95c2-0548a86064e9/volumes" Jul 16 00:09:45.323651 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.323616 2551 generic.go:356] "Generic (PLEG): container finished" podID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" containerID="43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224" exitCode=0 Jul 16 00:09:45.323850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.323670 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" Jul 16 00:09:45.323850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.323697 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" event={"ID":"fd70d526-84bf-48cc-94d3-2d4c64ad41c2","Type":"ContainerDied","Data":"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224"} Jul 16 00:09:45.323850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.323744 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m" event={"ID":"fd70d526-84bf-48cc-94d3-2d4c64ad41c2","Type":"ContainerDied","Data":"30e1923efc9c3c7001c7975769df9607ccf29799940b64b8d8152d134959dcc6"} Jul 16 00:09:45.323850 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.323769 2551 scope.go:117] "RemoveContainer" containerID="43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224" Jul 16 00:09:45.325962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.325795 2551 generic.go:356] "Generic (PLEG): container finished" podID="6df60380-09d0-4e51-aa7b-cd905be23c87" containerID="f3cc6a7ba39eb6eac269f7cb3cf94275bb4d3e975b98ad55c6f4cf110b5c682d" exitCode=0 Jul 16 00:09:45.325962 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.325927 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644606116176" event={"ID":"6df60380-09d0-4e51-aa7b-cd905be23c87","Type":"ContainerDied","Data":"f3cc6a7ba39eb6eac269f7cb3cf94275bb4d3e975b98ad55c6f4cf110b5c682d"} Jul 16 00:09:45.333144 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.333109 2551 scope.go:117] "RemoveContainer" containerID="43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224" Jul 16 00:09:45.333503 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:45.333480 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224\": container with ID starting with 43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224 not found: ID does not exist" containerID="43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224" Jul 16 00:09:45.333580 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.333515 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224"} err="failed to get container status \"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224\": rpc error: code = NotFound desc = could not find container \"43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224\": container with ID starting with 43c583a587132240fde82b6ad59f737de3158fe135122f14ce2c76c18b420224 not found: ID does not exist" Jul 16 00:09:45.354395 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.354356 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:09:45.358380 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:45.358345 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-079998/rate-test-model-079998-kserve-75bd66bff5-n5d8m"] Jul 16 00:09:46.485648 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.485611 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:46.507364 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.507334 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644606116176_6df60380-09d0-4e51-aa7b-cd905be23c87/test-curl-7487-139644606116176/0.log" Jul 16 00:09:46.530979 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.530939 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:46.533199 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.533169 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:46.623701 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.623661 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kr2t\" (UniqueName: \"kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t\") pod \"6df60380-09d0-4e51-aa7b-cd905be23c87\" (UID: \"6df60380-09d0-4e51-aa7b-cd905be23c87\") " Jul 16 00:09:46.625807 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.625772 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t" (OuterVolumeSpecName: "kube-api-access-4kr2t") pod "6df60380-09d0-4e51-aa7b-cd905be23c87" (UID: "6df60380-09d0-4e51-aa7b-cd905be23c87"). InnerVolumeSpecName "kube-api-access-4kr2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:46.654185 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654143 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:46.654531 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654519 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" containerName="main" Jul 16 00:09:46.654573 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654533 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" containerName="main" Jul 16 00:09:46.654573 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654549 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df60380-09d0-4e51-aa7b-cd905be23c87" containerName="test-curl-7487-139644606116176" Jul 16 00:09:46.654573 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654555 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df60380-09d0-4e51-aa7b-cd905be23c87" containerName="test-curl-7487-139644606116176" Jul 16 00:09:46.654668 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654613 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" containerName="main" Jul 16 00:09:46.654668 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.654620 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df60380-09d0-4e51-aa7b-cd905be23c87" containerName="test-curl-7487-139644606116176" Jul 16 00:09:46.657732 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.657706 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:46.663415 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.663383 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:46.724927 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.724879 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxck\" (UniqueName: \"kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck\") pod \"test-curl-7487-139644606116176\" (UID: \"f34b0c08-839c-4b24-a9a8-99f99f7f570b\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:46.725111 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.725042 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kr2t\" (UniqueName: \"kubernetes.io/projected/6df60380-09d0-4e51-aa7b-cd905be23c87-kube-api-access-4kr2t\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:46.826226 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.826185 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxck\" (UniqueName: \"kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck\") pod \"test-curl-7487-139644606116176\" (UID: \"f34b0c08-839c-4b24-a9a8-99f99f7f570b\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:46.834082 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.834049 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxck\" (UniqueName: \"kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck\") pod \"test-curl-7487-139644606116176\" (UID: \"f34b0c08-839c-4b24-a9a8-99f99f7f570b\") " pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:46.968856 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:46.968767 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:47.112937 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.112902 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:47.114832 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:09:47.114803 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf34b0c08_839c_4b24_a9a8_99f99f7f570b.slice/crio-0507077bdb367fdbe659a6db976f1120152654f6f306075c9cff293df2d2da73 WatchSource:0}: Error finding container 0507077bdb367fdbe659a6db976f1120152654f6f306075c9cff293df2d2da73: Status 404 returned error can't find the container with id 0507077bdb367fdbe659a6db976f1120152654f6f306075c9cff293df2d2da73 Jul 16 00:09:47.304346 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.304308 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df60380-09d0-4e51-aa7b-cd905be23c87" path="/var/lib/kubelet/pods/6df60380-09d0-4e51-aa7b-cd905be23c87/volumes" Jul 16 00:09:47.304736 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.304719 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd70d526-84bf-48cc-94d3-2d4c64ad41c2" path="/var/lib/kubelet/pods/fd70d526-84bf-48cc-94d3-2d4c64ad41c2/volumes" Jul 16 00:09:47.341319 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.341279 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644606116176" event={"ID":"f34b0c08-839c-4b24-a9a8-99f99f7f570b","Type":"ContainerStarted","Data":"0507077bdb367fdbe659a6db976f1120152654f6f306075c9cff293df2d2da73"} Jul 16 00:09:47.342770 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.342744 2551 scope.go:117] "RemoveContainer" containerID="f3cc6a7ba39eb6eac269f7cb3cf94275bb4d3e975b98ad55c6f4cf110b5c682d" Jul 16 00:09:47.342770 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:47.342765 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:48.350516 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:48.350484 2551 generic.go:356] "Generic (PLEG): container finished" podID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" containerID="8c0ba47cb65264e1584a622e4b72891580d9cbd9ccae9019356e036c2a425bc0" exitCode=0 Jul 16 00:09:48.350974 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:48.350585 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7487-139644606116176" event={"ID":"f34b0c08-839c-4b24-a9a8-99f99f7f570b","Type":"ContainerDied","Data":"8c0ba47cb65264e1584a622e4b72891580d9cbd9ccae9019356e036c2a425bc0"} Jul 16 00:09:49.505538 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.505501 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:49.524925 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.524887 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7487-139644606116176_f34b0c08-839c-4b24-a9a8-99f99f7f570b/test-curl-7487-139644606116176/0.log" Jul 16 00:09:49.547203 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.547164 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:49.550468 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.550430 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7487-139644606116176"] Jul 16 00:09:49.652226 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.652101 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzxck\" (UniqueName: \"kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck\") pod \"f34b0c08-839c-4b24-a9a8-99f99f7f570b\" (UID: \"f34b0c08-839c-4b24-a9a8-99f99f7f570b\") " Jul 16 00:09:49.654303 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.654273 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck" (OuterVolumeSpecName: "kube-api-access-mzxck") pod "f34b0c08-839c-4b24-a9a8-99f99f7f570b" (UID: "f34b0c08-839c-4b24-a9a8-99f99f7f570b"). InnerVolumeSpecName "kube-api-access-mzxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:09:49.752999 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:49.752953 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzxck\" (UniqueName: \"kubernetes.io/projected/f34b0c08-839c-4b24-a9a8-99f99f7f570b-kube-api-access-mzxck\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:09:50.363198 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:50.363160 2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0507077bdb367fdbe659a6db976f1120152654f6f306075c9cff293df2d2da73" Jul 16 00:09:50.363198 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:50.363177 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:50.373989 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:50.373955 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644606116176\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:51.303424 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:09:51.303378 2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7487-139644606116176\" is forbidden: User \"system:node:ip-10-0-142-149.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-149.ec2.internal' and this object" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" pod="opendatahub/test-curl-7487-139644606116176" Jul 16 00:09:51.304732 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:09:51.304703 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" path="/var/lib/kubelet/pods/f34b0c08-839c-4b24-a9a8-99f99f7f570b/volumes" Jul 16 00:10:00.196360 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:00.196318 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:10:02.127878 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:02.127839 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:10:02.128389 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:02.128103 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" containerID="cri-o://960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a" gracePeriod=30 Jul 16 00:10:04.305170 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:04.305098 2551 patch_prober.go:28] interesting pod/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 00:10:04.305586 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:04.305185 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 00:10:07.398446 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.398362 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:10:07.459627 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.459594 2551 generic.go:356] "Generic (PLEG): container finished" podID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerID="960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a" exitCode=0 Jul 16 00:10:07.459823 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.459637 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerDied","Data":"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a"} Jul 16 00:10:07.459823 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.459642 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" Jul 16 00:10:07.459823 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.459684 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb" event={"ID":"3630454c-eff4-460d-9213-7c9cdc0c0881","Type":"ContainerDied","Data":"349d3083f562e0a3f16f181559dd824320ba035aca07d7748a31c63727415a00"} Jul 16 00:10:07.459823 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.459707 2551 scope.go:117] "RemoveContainer" containerID="960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a" Jul 16 00:10:07.469078 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.469056 2551 scope.go:117] "RemoveContainer" containerID="dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564" Jul 16 00:10:07.478134 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.478099 2551 scope.go:117] "RemoveContainer" containerID="960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a" Jul 16 00:10:07.478498 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:10:07.478476 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a\": container with ID starting with 960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a not found: ID does not exist" containerID="960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a" Jul 16 00:10:07.478597 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.478506 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a"} err="failed to get container status \"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a\": rpc error: code = NotFound desc = could not find container \"960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a\": container with ID starting with 960af76b6a35ce4aaa754c0b349e085417466e9f30c2717a2bf1933bafa6326a not found: ID does not exist" Jul 16 00:10:07.478597 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.478524 2551 scope.go:117] "RemoveContainer" containerID="dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564" Jul 16 00:10:07.478782 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:10:07.478761 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564\": container with ID starting with dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564 not found: ID does not exist" containerID="dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564" Jul 16 00:10:07.478841 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.478793 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564"} err="failed to get container status \"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564\": rpc error: code = NotFound desc = could not find container \"dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564\": container with ID starting with dc6129878b7b7fb60429fb6ed23d8e007e4d2ae519f73e43d6e574e05f185564 not found: ID does not exist" Jul 16 00:10:07.516169 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516104 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516183 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516204 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516235 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516260 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516285 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62zkf\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516342 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516320 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516380 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516422 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data\") pod \"3630454c-eff4-460d-9213-7c9cdc0c0881\" (UID: \"3630454c-eff4-460d-9213-7c9cdc0c0881\") " Jul 16 00:10:07.516653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516561 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:07.516653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516571 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:07.516866 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516683 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.516866 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516707 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.516866 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516788 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:07.517021 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.516872 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:10:07.517393 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.517372 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data" (OuterVolumeSpecName: "istio-data") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:07.518530 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.518504 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:10:07.518638 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.518605 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:07.518922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.518887 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token" (OuterVolumeSpecName: "istio-token") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:10:07.518922 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.518897 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf" (OuterVolumeSpecName: "kube-api-access-62zkf") pod "3630454c-eff4-460d-9213-7c9cdc0c0881" (UID: "3630454c-eff4-460d-9213-7c9cdc0c0881"). InnerVolumeSpecName "kube-api-access-62zkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:10:07.617788 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617745 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.617788 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617778 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.617788 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617788 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.617788 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617797 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zkf\" (UniqueName: \"kubernetes.io/projected/3630454c-eff4-460d-9213-7c9cdc0c0881-kube-api-access-62zkf\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.618077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617807 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3630454c-eff4-460d-9213-7c9cdc0c0881-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.618077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617816 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.618077 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.617824 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3630454c-eff4-460d-9213-7c9cdc0c0881-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:07.788877 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.788834 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:10:07.792866 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:07.792834 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-079998-openshift-default-8c8f6bf59-v42rb"] Jul 16 00:10:09.306288 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:09.306248 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" path="/var/lib/kubelet/pods/3630454c-eff4-460d-9213-7c9cdc0c0881/volumes" Jul 16 00:10:12.084495 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:12.084443 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ql6l2"] Jul 16 00:10:14.533532 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:14.533486 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:10:14.533941 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:14.533779 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" containerID="cri-o://a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795" gracePeriod=30 Jul 16 00:10:15.188420 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:15.188388 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9cbfc9bb-wkpxp_2f6e3c18-44a7-462a-8516-4108314df640/authorino/0.log" Jul 16 00:10:19.789527 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.789492 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:10:19.820474 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820369 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820474 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820460 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820716 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820488 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js7vk\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820716 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820518 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820716 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820662 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820735 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820740 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820767 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820825 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820832 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.820893 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.820868 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs\") pod \"0c29d621-cad1-46d8-b2bc-6f1458b78705\" (UID: \"0c29d621-cad1-46d8-b2bc-6f1458b78705\") " Jul 16 00:10:19.821250 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821078 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 00:10:19.821361 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821335 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:19.821801 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821777 2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-credential-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.821801 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821806 2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c29d621-cad1-46d8-b2bc-6f1458b78705-istiod-ca-cert\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.821980 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821822 2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-socket\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.821980 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.821927 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:19.822981 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.822951 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:10:19.823094 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.823069 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk" (OuterVolumeSpecName: "kube-api-access-js7vk") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "kube-api-access-js7vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 00:10:19.823282 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.823255 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 00:10:19.823388 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.823258 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c29d621-cad1-46d8-b2bc-6f1458b78705" (UID: "0c29d621-cad1-46d8-b2bc-6f1458b78705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 00:10:19.922857 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922817 2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-token\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.922857 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922850 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7vk\" (UniqueName: \"kubernetes.io/projected/0c29d621-cad1-46d8-b2bc-6f1458b78705-kube-api-access-js7vk\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.922857 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922860 2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-podinfo\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.922857 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922869 2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-envoy\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.923178 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922877 2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-istio-data\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:19.923178 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:19.922885 2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c29d621-cad1-46d8-b2bc-6f1458b78705-workload-certs\") on node \"ip-10-0-142-149.ec2.internal\" DevicePath \"\"" Jul 16 00:10:20.513087 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.513049 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-47wnf_b6d0286f-b670-4834-b456-f1ca01006f82/manager/0.log" Jul 16 00:10:20.538389 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.538356 2551 generic.go:356] "Generic (PLEG): container finished" podID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerID="a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795" exitCode=0 Jul 16 00:10:20.538598 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.538435 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" Jul 16 00:10:20.538598 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.538452 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerDied","Data":"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795"} Jul 16 00:10:20.538598 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.538507 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv" event={"ID":"0c29d621-cad1-46d8-b2bc-6f1458b78705","Type":"ContainerDied","Data":"188a9dc97733261324994effb0d564f670306015ffcfd9732764a82ea13bcf09"} Jul 16 00:10:20.538598 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.538533 2551 scope.go:117] "RemoveContainer" containerID="a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795" Jul 16 00:10:20.551694 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.551664 2551 scope.go:117] "RemoveContainer" containerID="7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6" Jul 16 00:10:20.562194 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.562171 2551 scope.go:117] "RemoveContainer" containerID="a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795" Jul 16 00:10:20.562452 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.562430 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:10:20.562519 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:10:20.562498 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795\": container with ID starting with a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795 not found: ID does not exist" containerID="a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795" Jul 16 00:10:20.562566 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.562531 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795"} err="failed to get container status \"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795\": rpc error: code = NotFound desc = could not find container \"a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795\": container with ID starting with a6903a3b2d40a86af9893e0d9ef4abdeb3d096734d828ab82cad1c3871f03795 not found: ID does not exist" Jul 16 00:10:20.562566 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.562550 2551 scope.go:117] "RemoveContainer" containerID="7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6" Jul 16 00:10:20.562799 ip-10-0-142-149 kubenswrapper[2551]: E0716 00:10:20.562783 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6\": container with ID starting with 7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6 not found: ID does not exist" containerID="7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6" Jul 16 00:10:20.562852 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.562802 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6"} err="failed to get container status \"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6\": rpc error: code = NotFound desc = could not find container \"7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6\": container with ID starting with 7b99ff667da6b6b9a30a2cd284a001df79d26887d575360c8a31fa2b9e1b32d6 not found: ID does not exist" Jul 16 00:10:20.564484 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.564458 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-91d115-openshift-default-5d84b48f47-lb5cv"] Jul 16 00:10:20.634447 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.634415 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-49m85_c8513607-5504-45da-8a33-24b47fc86f6c/manager/0.log" Jul 16 00:10:20.885234 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:20.885091 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-78f9q_1af7a9c9-f783-4f73-9b67-a869f4761843/server/0.log" Jul 16 00:10:21.003586 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:21.003548 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b2mrj_671e0999-eb83-489f-908d-875529d5668e/manager/0.log" Jul 16 00:10:21.115443 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:21.115419 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw_930c319f-8335-460c-80a2-fcd4d6cec804/copy-manifests/0.log" Jul 16 00:10:21.128399 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:21.128358 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw_930c319f-8335-460c-80a2-fcd4d6cec804/manager/0.log" Jul 16 00:10:21.303033 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:21.302977 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" path="/var/lib/kubelet/pods/0c29d621-cad1-46d8-b2bc-6f1458b78705/volumes" Jul 16 00:10:22.825482 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.825438 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/util/0.log" Jul 16 00:10:22.832873 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.832839 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/pull/0.log" Jul 16 00:10:22.839691 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.839669 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/extract/0.log" Jul 16 00:10:22.957686 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.957656 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/pull/0.log" Jul 16 00:10:22.964366 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.964344 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/extract/0.log" Jul 16 00:10:22.970874 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:22.970854 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/util/0.log" Jul 16 00:10:23.079607 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.079489 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/extract/0.log" Jul 16 00:10:23.086725 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.086699 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/util/0.log" Jul 16 00:10:23.093861 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.093828 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/pull/0.log" Jul 16 00:10:23.208264 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.208237 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/util/0.log" Jul 16 00:10:23.215840 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.215820 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/pull/0.log" Jul 16 00:10:23.223607 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.223583 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/extract/0.log" Jul 16 00:10:23.336943 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.336847 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9cbfc9bb-wkpxp_2f6e3c18-44a7-462a-8516-4108314df640/authorino/0.log" Jul 16 00:10:23.462801 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.462763 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bpds8_d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd/manager/0.log" Jul 16 00:10:23.802648 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:23.802607 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-98j7l_1907ec38-9bd2-4434-8c12-09d34406d019/registry-server/0.log" Jul 16 00:10:24.057148 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:24.057054 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ql6l2_8a67338b-2730-49d4-b97d-2b5d444696d1/limitador/0.log" Jul 16 00:10:24.173862 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:24.173819 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dtsw7_05a7ad7a-92c3-4778-9ed2-cf28db3df85a/manager/0.log" Jul 16 00:10:24.759015 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:24.758974 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bddb6d9b-9bf4t_0c14bfa3-4cfd-444b-a4a0-05c05dcc5989/kube-auth-proxy/0.log" Jul 16 00:10:26.020024 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:26.019981 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbkhmpn_3887d43c-ce7a-4f93-a05c-78e853c2af88/main/0.log" Jul 16 00:10:26.131914 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:26.131867 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cclqw_d7f1faa0-9677-46d6-8b56-c93e873be608/main/0.log" Jul 16 00:10:26.241572 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:26.241538 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-qmt6s_fb534609-c1bb-430a-830c-5f9030d66cc4/main/0.log" Jul 16 00:10:32.079745 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.079705 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-66nch/must-gather-6w79g"] Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080036 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080047 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080056 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080061 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080073 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080079 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080085 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" containerName="test-curl-7487-139644606116176" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080091 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" containerName="test-curl-7487-139644606116176" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080099 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080104 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080206 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080214 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34b0c08-839c-4b24-a9a8-99f99f7f570b" containerName="test-curl-7487-139644606116176" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080226 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080227 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080234 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c29d621-cad1-46d8-b2bc-6f1458b78705" containerName="istio-proxy" Jul 16 00:10:32.080868 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.080398 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3630454c-eff4-460d-9213-7c9cdc0c0881" containerName="istio-proxy" Jul 16 00:10:32.084910 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.084884 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.087340 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.087312 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66nch\"/\"openshift-service-ca.crt\"" Jul 16 00:10:32.089222 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.088483 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-66nch\"/\"kube-root-ca.crt\"" Jul 16 00:10:32.089222 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.088731 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-66nch\"/\"default-dockercfg-kjq8g\"" Jul 16 00:10:32.091158 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.091120 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66nch/must-gather-6w79g"] Jul 16 00:10:32.232782 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.232744 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-must-gather-output\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.233001 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.232792 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmd7k\" (UniqueName: \"kubernetes.io/projected/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-kube-api-access-vmd7k\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.333588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.333491 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-must-gather-output\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.333588 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.333534 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmd7k\" (UniqueName: \"kubernetes.io/projected/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-kube-api-access-vmd7k\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.333877 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.333856 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-must-gather-output\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.341693 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.341658 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmd7k\" (UniqueName: \"kubernetes.io/projected/fa10fbf4-4282-49bf-a1f6-0b6365f31d16-kube-api-access-vmd7k\") pod \"must-gather-6w79g\" (UID: \"fa10fbf4-4282-49bf-a1f6-0b6365f31d16\") " pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.396678 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.396638 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66nch/must-gather-6w79g" Jul 16 00:10:32.545229 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.545199 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66nch/must-gather-6w79g"] Jul 16 00:10:32.603837 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:32.603749 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66nch/must-gather-6w79g" event={"ID":"fa10fbf4-4282-49bf-a1f6-0b6365f31d16","Type":"ContainerStarted","Data":"a43f8ee0c05db529aea7801eaf279c710e905180839191917f898b41beb22dd0"} Jul 16 00:10:33.612624 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:33.612591 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66nch/must-gather-6w79g" event={"ID":"fa10fbf4-4282-49bf-a1f6-0b6365f31d16","Type":"ContainerStarted","Data":"baaa7f3a5b97d3a3f6cdde7943ab9b6a1183d806a26d143674e454278fd12d76"} Jul 16 00:10:34.623333 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:34.623280 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66nch/must-gather-6w79g" event={"ID":"fa10fbf4-4282-49bf-a1f6-0b6365f31d16","Type":"ContainerStarted","Data":"457c5bbbe7f142942183578ececf282a1a61f037a6c6717c57bad8479361eaff"} Jul 16 00:10:34.641567 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:34.641495 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-66nch/must-gather-6w79g" podStartSLOduration=1.683001141 podStartE2EDuration="2.641472144s" totalImagesPullingTime="958.471003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 00:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:10:34.638355914 +0000 UTC m=+2423.927171343" watchObservedRunningTime="2026-07-16 00:10:34.641472144 +0000 UTC m=+2423.930287574" Jul 16 00:10:35.132705 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:35.132668 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x5ftd_6532cecb-fc83-485e-958e-f98fccb16ae0/global-pull-secret-syncer/0.log" Jul 16 00:10:35.216362 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:35.216333 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rlzmg_41520bf2-b9f5-44e2-8c5b-0f497e79fa13/konnectivity-agent/0.log" Jul 16 00:10:35.259537 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:35.259503 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-149.ec2.internal_21088ddecb53e5296158fe55990a64c7/haproxy/0.log" Jul 16 00:10:40.005501 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.005404 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/extract/0.log" Jul 16 00:10:40.029049 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.029001 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/util/0.log" Jul 16 00:10:40.062197 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.062172 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759npr2p_0d5622e5-706f-4405-b8bd-b5045c7fe0a4/pull/0.log" Jul 16 00:10:40.091220 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.091186 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/extract/0.log" Jul 16 00:10:40.117768 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.117729 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/util/0.log" Jul 16 00:10:40.138821 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.138790 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0s5zql_7334b077-8fd1-4485-bd23-b5b00990e4b0/pull/0.log" Jul 16 00:10:40.165675 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.165627 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/extract/0.log" Jul 16 00:10:40.188510 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.188473 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/util/0.log" Jul 16 00:10:40.212956 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.212919 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736tgs5_da2931f9-bfc0-44a5-9c67-37642969172b/pull/0.log" Jul 16 00:10:40.242682 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.242590 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/extract/0.log" Jul 16 00:10:40.266463 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.266370 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/util/0.log" Jul 16 00:10:40.288301 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.288264 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sdrcg_4d761059-19c1-4239-b837-b73e3ecb0e7c/pull/0.log" Jul 16 00:10:40.731661 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.731577 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b9cbfc9bb-wkpxp_2f6e3c18-44a7-462a-8516-4108314df640/authorino/0.log" Jul 16 00:10:40.774067 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.774029 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bpds8_d437fb2a-a7ba-4f17-a5e6-dd2e2b8e16fd/manager/0.log" Jul 16 00:10:40.892468 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:40.892427 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-98j7l_1907ec38-9bd2-4434-8c12-09d34406d019/registry-server/0.log" Jul 16 00:10:41.019781 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:41.019748 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ql6l2_8a67338b-2730-49d4-b97d-2b5d444696d1/limitador/0.log" Jul 16 00:10:41.099677 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:41.099632 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dtsw7_05a7ad7a-92c3-4778-9ed2-cf28db3df85a/manager/0.log" Jul 16 00:10:43.064165 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:43.064109 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbbb9_09d68cb9-c524-481e-912f-2a14ed2d9327/node-exporter/0.log" Jul 16 00:10:43.086356 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:43.086323 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbbb9_09d68cb9-c524-481e-912f-2a14ed2d9327/kube-rbac-proxy/0.log" Jul 16 00:10:43.113434 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:43.113404 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cbbb9_09d68cb9-c524-481e-912f-2a14ed2d9327/init-textfile/0.log" Jul 16 00:10:45.586012 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.585974 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb"] Jul 16 00:10:45.591604 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.591577 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.599044 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.598996 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb"] Jul 16 00:10:45.773349 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.773306 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmf4\" (UniqueName: \"kubernetes.io/projected/c3738f1c-e965-42f1-872a-643265ead762-kube-api-access-nbmf4\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.773653 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.773628 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-lib-modules\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.773882 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.773856 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-podres\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.774004 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.773988 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-sys\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.774086 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.774041 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-proc\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.874966 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.874868 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-lib-modules\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.874966 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.874916 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-podres\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.874970 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-sys\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.874987 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-proc\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.875087 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-sys\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.875112 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-podres\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.875165 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-lib-modules\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.875182 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmf4\" (UniqueName: \"kubernetes.io/projected/c3738f1c-e965-42f1-872a-643265ead762-kube-api-access-nbmf4\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.875243 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.875190 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c3738f1c-e965-42f1-872a-643265ead762-proc\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.883070 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.883037 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmf4\" (UniqueName: \"kubernetes.io/projected/c3738f1c-e965-42f1-872a-643265ead762-kube-api-access-nbmf4\") pod \"perf-node-gather-daemonset-2mzkb\" (UID: \"c3738f1c-e965-42f1-872a-643265ead762\") " pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:45.909596 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:45.909555 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:46.113964 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:46.113495 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb"] Jul 16 00:10:46.119998 ip-10-0-142-149 kubenswrapper[2551]: W0716 00:10:46.119957 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc3738f1c_e965_42f1_872a_643265ead762.slice/crio-d17a3b434c18f7c4115c2e03845963b0f848249b47928e4136a940922cabbd51 WatchSource:0}: Error finding container d17a3b434c18f7c4115c2e03845963b0f848249b47928e4136a940922cabbd51: Status 404 returned error can't find the container with id d17a3b434c18f7c4115c2e03845963b0f848249b47928e4136a940922cabbd51 Jul 16 00:10:46.721776 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:46.721725 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" event={"ID":"c3738f1c-e965-42f1-872a-643265ead762","Type":"ContainerStarted","Data":"b361ab2cf1de092e5f3d8862ac0e604ede42867751656d87b7cacc64cbe435ba"} Jul 16 00:10:46.722279 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:46.721804 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" event={"ID":"c3738f1c-e965-42f1-872a-643265ead762","Type":"ContainerStarted","Data":"d17a3b434c18f7c4115c2e03845963b0f848249b47928e4136a940922cabbd51"} Jul 16 00:10:46.722279 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:46.721833 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:46.737318 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:46.737246 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" podStartSLOduration=1.7372224680000001 podStartE2EDuration="1.737222468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 00:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 00:10:46.736297206 +0000 UTC m=+2436.025112636" watchObservedRunningTime="2026-07-16 00:10:46.737222468 +0000 UTC m=+2436.026037900" Jul 16 00:10:47.841351 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:47.841318 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j294m_386eada3-b78e-4126-9a67-89fc6029278b/dns/0.log" Jul 16 00:10:47.862635 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:47.862609 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j294m_386eada3-b78e-4126-9a67-89fc6029278b/kube-rbac-proxy/0.log" Jul 16 00:10:47.962712 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:47.962667 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqc8n_f7b9d180-ba25-43e8-9ddf-7a91608ea469/dns-node-resolver/0.log" Jul 16 00:10:48.470576 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:48.470549 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29736000-m2s58_f5075908-2d0e-4db7-b2b2-6ff86b85e549/image-pruner/0.log" Jul 16 00:10:48.515144 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:48.515095 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-545dbd4fc7-pb425_6d303399-f322-4d8b-8555-514663a4740f/registry/0.log" Jul 16 00:10:48.553630 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:48.553604 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gj5mm_b8f925a3-e179-41e3-af2e-1eb891af39be/node-ca/0.log" Jul 16 00:10:49.711963 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:49.711923 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bddb6d9b-9bf4t_0c14bfa3-4cfd-444b-a4a0-05c05dcc5989/kube-auth-proxy/0.log" Jul 16 00:10:50.545842 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:50.545816 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mt28x_638213a2-5e50-4ea6-a930-1b4eea3f7058/serve-healthcheck-canary/0.log" Jul 16 00:10:51.062050 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:51.062017 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m2p6_aae883d1-fd1b-4c3b-8a85-3417ff6e93f7/kube-rbac-proxy/0.log" Jul 16 00:10:51.083960 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:51.083934 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m2p6_aae883d1-fd1b-4c3b-8a85-3417ff6e93f7/exporter/0.log" Jul 16 00:10:51.105080 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:51.105012 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m2p6_aae883d1-fd1b-4c3b-8a85-3417ff6e93f7/extractor/0.log" Jul 16 00:10:52.740065 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:52.739920 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-66nch/perf-node-gather-daemonset-2mzkb" Jul 16 00:10:53.340555 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.340524 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-47wnf_b6d0286f-b670-4834-b456-f1ca01006f82/manager/0.log" Jul 16 00:10:53.505305 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.505269 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-49m85_c8513607-5504-45da-8a33-24b47fc86f6c/manager/0.log" Jul 16 00:10:53.673428 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.673336 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-78f9q_1af7a9c9-f783-4f73-9b67-a869f4761843/server/0.log" Jul 16 00:10:53.752285 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.752244 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-b2mrj_671e0999-eb83-489f-908d-875529d5668e/manager/0.log" Jul 16 00:10:53.839855 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.839814 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw_930c319f-8335-460c-80a2-fcd4d6cec804/manager/0.log" Jul 16 00:10:53.858577 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:53.858545 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6ccf6fdd4-m5wmw_930c319f-8335-460c-80a2-fcd4d6cec804/copy-manifests/0.log" Jul 16 00:10:55.530158 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:55.530105 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fbxss_38925f12-408a-40a7-be41-4fe5ff3907f0/openshift-lws-operator/0.log" Jul 16 00:10:57.122260 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:57.122227 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bddb6d9b-9bf4t_0c14bfa3-4cfd-444b-a4a0-05c05dcc5989/kube-auth-proxy/0.log" Jul 16 00:10:58.728454 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:10:58.728420 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bddb6d9b-9bf4t_0c14bfa3-4cfd-444b-a4a0-05c05dcc5989/kube-auth-proxy/0.log" Jul 16 00:11:02.196188 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.196156 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/kube-multus-additional-cni-plugins/0.log" Jul 16 00:11:02.216743 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.216715 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/egress-router-binary-copy/0.log" Jul 16 00:11:02.236946 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.236918 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/cni-plugins/0.log" Jul 16 00:11:02.256164 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.256114 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/bond-cni-plugin/0.log" Jul 16 00:11:02.277997 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.277966 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/routeoverride-cni/0.log" Jul 16 00:11:02.297198 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.297170 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/whereabouts-cni-bincopy/0.log" Jul 16 00:11:02.316875 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.316845 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djqxk_3b47ef9e-9cbf-423d-b6e7-b50c9102cf91/whereabouts-cni/0.log" Jul 16 00:11:02.623054 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.623020 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nbfwm_d8b83ce7-2b96-4123-bae7-47993008f3b1/kube-multus/0.log" Jul 16 00:11:02.680221 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.680164 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mggsj_f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98/network-metrics-daemon/0.log" Jul 16 00:11:02.697989 ip-10-0-142-149 kubenswrapper[2551]: I0716 00:11:02.697957 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mggsj_f5303d4e-f2ae-4c0c-97c8-4340f4dc4f98/kube-rbac-proxy/0.log"