Jul 10 00:15:41.386771 ip-10-0-137-8 systemd[1]: Starting Kubernetes Kubelet... Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.853676 2548 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869313 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869327 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869332 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869336 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869339 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869343 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869346 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:41.871354 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869349 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869352 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869355 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869358 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869360 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869364 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869367 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869370 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869372 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869375 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869378 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869382 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869386 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869390 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869393 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869396 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869399 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869402 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869405 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:41.875283 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869408 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869410 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869414 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869416 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869419 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869421 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869424 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869426 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869430 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869432 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869435 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869438 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869441 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869444 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869447 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869449 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869452 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869455 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869457 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869461 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:41.878187 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869464 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869466 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869469 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869472 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869474 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869478 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869482 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869485 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869488 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869506 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869515 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869518 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869521 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869524 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869527 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869530 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869532 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869535 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869538 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:41.881868 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869540 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869543 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869547 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869550 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869552 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869555 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869558 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869561 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869563 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869566 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869569 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869571 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869575 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869579 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869582 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869584 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869587 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869590 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869592 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869595 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:41.894005 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869597 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869600 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869602 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869605 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869607 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869610 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869613 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869616 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869619 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869621 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869624 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869626 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869629 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.869631 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871864 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871873 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871878 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871881 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871883 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871886 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:41.897461 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871889 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871891 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871894 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871896 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871901 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871904 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871907 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871910 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871912 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871915 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871917 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871920 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871923 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871925 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871928 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871930 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871933 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871935 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871938 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871941 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:41.900928 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871943 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871946 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871949 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871952 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871954 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871957 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871959 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871962 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871965 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871969 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871973 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871975 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871978 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871982 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871986 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871989 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871992 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871995 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.871998 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:41.905784 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872001 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872004 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872006 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872009 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872012 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872017 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872020 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872023 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872026 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872028 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872031 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872034 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872036 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872039 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872042 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872045 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872048 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872050 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872052 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872055 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:41.913671 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872057 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872060 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872063 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872066 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872069 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872071 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872074 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872076 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872079 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872082 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872085 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872087 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872090 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872094 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872096 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872099 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872101 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872104 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872107 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872109 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:41.918527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872111 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872114 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872117 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872120 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872122 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872125 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872128 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872130 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872133 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872135 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872138 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872141 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872143 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872146 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872559 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872565 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872568 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872571 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872573 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:41.926719 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872576 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872579 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872582 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872584 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872587 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872590 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872592 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872596 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872599 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872602 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872605 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872607 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872610 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872612 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872615 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872617 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872620 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872622 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872625 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872627 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:41.931503 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872630 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872633 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872635 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872638 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872641 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872643 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872646 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872648 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872651 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872654 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872656 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872659 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872663 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872666 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872669 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872672 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872675 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872677 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872683 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872686 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:41.939395 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872689 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872691 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872694 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872696 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872699 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872701 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872704 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872706 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872709 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872711 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872714 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872717 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872720 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872722 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872726 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872728 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872731 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872733 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872736 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:41.944430 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872739 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:41.944895 ip-10-0-137-8 systemd[1]: Started Kubernetes Kubelet. Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872742 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872745 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872748 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872754 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872756 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872759 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872769 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872773 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872776 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872778 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872783 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872785 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872788 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872790 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872793 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872796 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872798 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872801 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872803 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:41.954995 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872806 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872811 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872814 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872818 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872820 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872823 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872826 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872829 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872832 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872835 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872837 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872840 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872843 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872846 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.872848 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872933 2548 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872953 2548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872960 2548 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872965 2548 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872969 2548 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872972 2548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 00:15:41.963794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872976 2548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872981 2548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872985 2548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872988 2548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872992 2548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872995 2548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.872999 2548 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873002 2548 flags.go:64] FLAG: --cgroup-root="" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873006 2548 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873009 2548 flags.go:64] FLAG: --client-ca-file="" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873011 2548 flags.go:64] FLAG: --cloud-provider="external" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873014 2548 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873020 2548 flags.go:64] FLAG: --cluster-domain="" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873023 2548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873026 2548 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873029 2548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873032 2548 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873036 2548 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873039 2548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873042 2548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873046 2548 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873049 2548 flags.go:64] FLAG: --contention-profiling="false" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873052 2548 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873055 2548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 00:15:41.972848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873058 2548 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873061 2548 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873065 2548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873070 2548 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873073 2548 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873076 2548 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873079 2548 flags.go:64] FLAG: --enable-server="true" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873082 2548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873086 2548 flags.go:64] FLAG: --event-burst="100" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873089 2548 flags.go:64] FLAG: --event-qps="50" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873094 2548 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873097 2548 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873100 2548 flags.go:64] FLAG: --eviction-hard="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873104 2548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873107 2548 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873110 2548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873112 2548 flags.go:64] FLAG: --eviction-soft="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873115 2548 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873118 2548 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873121 2548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873124 2548 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873127 2548 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873130 2548 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873133 2548 flags.go:64] FLAG: --feature-gates="" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873137 2548 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 00:15:41.978384 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873140 2548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873143 2548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873146 2548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873149 2548 flags.go:64] FLAG: --healthz-port="10248" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873153 2548 flags.go:64] FLAG: --help="false" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873156 2548 flags.go:64] FLAG: --hostname-override="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873159 2548 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873162 2548 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873165 2548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873168 2548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873173 2548 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873175 2548 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873179 2548 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873182 2548 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873185 2548 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873188 2548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873191 2548 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873195 2548 flags.go:64] FLAG: --kube-reserved="" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873198 2548 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873201 2548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873204 2548 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873207 2548 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873210 2548 flags.go:64] FLAG: --lock-file="" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873213 2548 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 00:15:42.155935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873216 2548 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873219 2548 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873225 2548 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873228 2548 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873231 2548 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873233 2548 flags.go:64] FLAG: --logging-format="text" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873236 2548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873240 2548 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873242 2548 flags.go:64] FLAG: --manifest-url="" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873245 2548 flags.go:64] FLAG: --manifest-url-header="" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873250 2548 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873253 2548 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873257 2548 flags.go:64] FLAG: --max-pods="110" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873261 2548 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873264 2548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873267 2548 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873269 2548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873272 2548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873277 2548 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873280 2548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873287 2548 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873290 2548 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873293 2548 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 00:15:42.156944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873296 2548 flags.go:64] FLAG: --pod-cidr="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873299 2548 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873306 2548 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873309 2548 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873313 2548 flags.go:64] FLAG: --pods-per-core="0" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873316 2548 flags.go:64] FLAG: --port="10250" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873319 2548 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873322 2548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a3793b4d8d81450" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873325 2548 flags.go:64] FLAG: --qos-reserved="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873329 2548 flags.go:64] FLAG: --read-only-port="10255" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873332 2548 flags.go:64] FLAG: --register-node="true" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873334 2548 flags.go:64] FLAG: --register-with-taints="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873338 2548 flags.go:64] FLAG: --registry-burst="10" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873341 2548 flags.go:64] FLAG: --registry-qps="5" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873344 2548 flags.go:64] FLAG: --reserved-cpus="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873347 2548 flags.go:64] FLAG: --reserved-memory="" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873350 2548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873353 2548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873356 2548 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873359 2548 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873362 2548 flags.go:64] FLAG: --runonce="false" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873365 2548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873368 2548 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873371 2548 flags.go:64] FLAG: --seccomp-default="false" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873374 2548 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 00:15:42.157907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873377 2548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873380 2548 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873383 2548 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873387 2548 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873390 2548 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873393 2548 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873396 2548 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873399 2548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873402 2548 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873406 2548 flags.go:64] FLAG: --system-cgroups="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873409 2548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873414 2548 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873417 2548 flags.go:64] FLAG: --tls-cert-file="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873420 2548 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873425 2548 flags.go:64] FLAG: --tls-min-version="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873428 2548 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873431 2548 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873434 2548 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873436 2548 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873440 2548 flags.go:64] FLAG: --v="2" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873443 2548 flags.go:64] FLAG: --version="false" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873448 2548 flags.go:64] FLAG: --vmodule="" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873452 2548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873455 2548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873590 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:42.158884 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873595 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873599 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873602 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873605 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873608 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873611 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873614 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873616 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873619 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873621 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873625 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873628 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873630 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873633 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873635 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873638 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873642 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873645 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873648 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:42.161954 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873651 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873655 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873658 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873661 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873665 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873668 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873671 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873674 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873676 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873679 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873681 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873684 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873686 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873689 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873692 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873694 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873697 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873699 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873702 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:42.164544 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873704 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873707 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873710 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873712 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873716 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873719 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873721 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873724 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873726 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873729 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873734 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873737 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873740 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873745 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873748 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873751 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873754 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873757 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873760 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:42.169157 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873763 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873766 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873768 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873771 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873773 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873776 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873778 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873781 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873783 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873786 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873789 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873791 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873794 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873797 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873799 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873802 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873805 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873808 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873812 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873814 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:42.171223 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873817 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873819 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873822 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873827 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873829 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873832 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873834 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873837 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873840 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873842 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873845 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873847 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873850 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873852 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873855 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873857 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873860 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873862 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873865 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873867 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:42.173179 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.873870 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.873876 2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.885441 2548 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.885457 2548 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.885479 2548 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.885502 2548 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885566 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885573 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885577 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885580 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885583 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885586 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885589 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885592 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885595 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:42.174024 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885597 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885601 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885604 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885607 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885611 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885615 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885617 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885620 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885622 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885625 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885628 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885631 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885635 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885640 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885644 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885649 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885652 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885655 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885659 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885662 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:42.176540 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885664 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885667 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885670 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885672 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885675 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885678 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885682 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885686 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885689 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885692 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885695 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885698 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885701 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885706 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885709 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885714 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885718 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885723 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885727 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:42.177348 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885729 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885732 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885735 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885738 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885741 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885743 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885746 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885749 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885752 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885755 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885758 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885761 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885763 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885766 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885769 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885771 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885774 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885777 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885779 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885782 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:42.178999 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885784 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885788 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885791 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885796 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885800 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885803 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885806 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885809 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885811 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885814 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885816 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885821 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885823 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885826 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885829 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885831 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885834 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885836 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885839 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885842 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:42.179814 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885844 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885847 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885849 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885852 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885854 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885857 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885859 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885861 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885865 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885869 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885873 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.885880 2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.885998 2548 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886004 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:15:42.181055 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886007 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886010 2548 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886013 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886016 2548 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886019 2548 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886023 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886028 2548 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886032 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886035 2548 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886041 2548 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886045 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886049 2548 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886051 2548 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886054 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886056 2548 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886059 2548 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886061 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886064 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886067 2548 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:15:42.181892 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886069 2548 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886072 2548 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886075 2548 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886077 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886080 2548 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886082 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886085 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886087 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886090 2548 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886093 2548 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886095 2548 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886099 2548 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886103 2548 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886107 2548 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886111 2548 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886114 2548 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886117 2548 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886120 2548 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886122 2548 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886126 2548 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:15:42.182709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886129 2548 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886131 2548 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886135 2548 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886137 2548 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886141 2548 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886143 2548 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886146 2548 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886148 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886151 2548 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886153 2548 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886156 2548 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886158 2548 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886161 2548 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886163 2548 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886166 2548 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886168 2548 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886171 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886174 2548 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886179 2548 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886183 2548 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:15:42.183527 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886187 2548 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886190 2548 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886192 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886195 2548 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886199 2548 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886202 2548 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886205 2548 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886208 2548 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886211 2548 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886214 2548 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886216 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886221 2548 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886223 2548 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886226 2548 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886230 2548 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886233 2548 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886236 2548 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886241 2548 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886244 2548 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886247 2548 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:15:42.184333 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886250 2548 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886255 2548 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886259 2548 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886263 2548 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886266 2548 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886269 2548 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886271 2548 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886274 2548 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886276 2548 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886279 2548 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886281 2548 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886284 2548 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886286 2548 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886289 2548 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886292 2548 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886294 2548 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886298 2548 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:15:42.185145 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:41.886301 2548 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.886307 2548 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.886440 2548 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.888716 2548 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.890411 2548 server.go:1019] "Starting client certificate rotation" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.890528 2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.890572 2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.893350 2548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.894512 2548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.906253 2548 log.go:25] "Validated CRI v1 runtime API" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.908943 2548 log.go:25] "Validated CRI v1 image API" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.910682 2548 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 00:15:42.185954 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.915618 2548 fs.go:135] Filesystem UUIDs: map[519e0875-c60c-4e6a-98c6-f0e74373b663:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8e82504b-6cbc-45f1-afd7-2eb3c942ef92:/dev/nvme0n1p3] Jul 10 00:15:42.186364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.915633 2548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 00:15:42.186364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.921748 2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.922165 2548 manager.go:216] Machine: {Timestamp:2026-07-10 00:15:41.920732756 +0000 UTC m=+0.400620296 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111358 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec232aebaaf8893a27d6984371befef5 SystemUUID:ec232aeb-aaf8-893a-27d6-984371befef5 BootID:70b52c31-31c9-48c2-bda4-d96eb3fb9af9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:73:d5:ef:db:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:73:d5:ef:db:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:64:c6:e2:8a:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.922275 2548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.922358 2548 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.926995 2548 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.927030 2548 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.927785 2548 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.927796 2548 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.927804 2548 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.928585 2548 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.930131 2548 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933437 2548 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933559 2548 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933586 2548 kubelet.go:486] "Attempting to sync node with API server" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933595 2548 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933606 2548 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933614 2548 kubelet.go:398] "Adding apiserver pod source" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933623 2548 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 00:15:42.190955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933947 2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.933968 2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.936296 2548 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.938798 2548 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.938825 2548 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.938991 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939003 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939009 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939015 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939024 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939030 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939689 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939697 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939705 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939710 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939719 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.939726 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.940672 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.940679 2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.941695 2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qmbdm" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.943792 2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.943883 2548 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.943978 2548 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.944108 2548 server.go:1268] "Started kubelet" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.944446 2548 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.944521 2548 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.945166 2548 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.947734 2548 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.950941 2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qmbdm" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.954976 2548 server.go:310] "Adding debug handlers to kubelet server" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.958779 2548 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.958786 2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.958852 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.959550 2548 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.963708 2548 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.963725 2548 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.963854 2548 reconstruct.go:97] "Volume reconstruction finished" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.963864 2548 reconciler.go:29] "Reconciler: start to sync state" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964126 2548 factory.go:55] Registering systemd factory Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.964131 2548 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964165 2548 factory.go:223] Registration of the systemd container factory successfully Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964471 2548 factory.go:153] Registering CRI-O factory Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964487 2548 factory.go:223] Registration of the crio container factory successfully Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964580 2548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964607 2548 factory.go:103] Registering Raw factory Jul 10 00:15:42.191802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.964619 2548 manager.go:1196] Started watching for new ooms in manager Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.963997 2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-8.ec2.internal.18c0c57116da2588 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-8.ec2.internal,UID:ip-10-0-137-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-8.ec2.internal,},FirstTimestamp:2026-07-10 00:15:41.944087944 +0000 UTC m=+0.423975483,LastTimestamp:2026-07-10 00:15:41.944087944 +0000 UTC m=+0.423975483,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-8.ec2.internal,}" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.972182 2548 manager.go:349] Starting recovery of all containers Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.983662 2548 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:41.986680 2548 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.988970 2548 manager.go:354] Recovery completed Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.993354 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995184 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995213 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995228 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995794 2548 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995804 2548 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995821 2548 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.995847 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996563 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996590 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996603 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996972 2548 policy_none.go:49] "None policy: Start" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996986 2548 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.996996 2548 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:41.997902 2548 policy_none.go:47] "Start" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029187 2548 manager.go:337] "Starting Device Plugin manager" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.029216 2548 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029225 2548 server.go:87] "Starting device plugin registration server" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029433 2548 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029445 2548 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029699 2548 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029783 2548 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.029795 2548 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.030122 2548 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.030173 2548 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.130366 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.132087 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.132119 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.132132 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.132160 2548 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.138600 2548 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.138621 2548 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-8.ec2.internal\": node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.149260 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.149626 2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.150983 2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.151045 2548 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 00:15:42.194289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.151082 2548 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 00:15:42.195551 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.151126 2548 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 00:15:42.195551 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.153777 2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.629803 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.629813 2548 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal"] Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.630068 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.631107 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.631139 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.631163 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.632494 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.632711 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.632747 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633322 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633337 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633349 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633359 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633369 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.633377 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.634636 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.634664 2548 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.635417 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.635447 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.635463 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.663428 2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.665617 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.665661 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.665715 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c365a80243b37665c2c8c1b5f01aa50a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"c365a80243b37665c2c8c1b5f01aa50a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.668070 2548 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.740486 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.731994 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.765865 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.765902 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.765924 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c365a80243b37665c2c8c1b5f01aa50a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"c365a80243b37665c2c8c1b5f01aa50a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.765956 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c365a80243b37665c2c8c1b5f01aa50a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"c365a80243b37665c2c8c1b5f01aa50a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.766002 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.766019 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47aa25279bcd0f53cb13fb27c7635d7d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"47aa25279bcd0f53cb13fb27c7635d7d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.858995 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.837451 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.942870 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:42.942815 2548 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 10 00:15:42.983918 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.983875 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.983918 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.983875 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 00:10:42 +0000 UTC" deadline="2028-01-23 03:45:35.059496808 +0000 UTC" Jul 10 00:15:42.984053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.983922 2548 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 00:15:42.984053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.983934 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13491h29m52.075570071s" Jul 10 00:15:42.984053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.983875 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:42.984182 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:42.984106 2548 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:15:43.010898 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.010823 2548 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:43.059272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.059239 2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 10 00:15:43.073205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.073182 2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:15:43.074921 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.074905 2548 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 10 00:15:43.084601 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.084580 2548 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:15:43.167338 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.167307 2548 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:43.552288 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:43.552251 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47aa25279bcd0f53cb13fb27c7635d7d.slice/crio-048b42f4d9fec22e382f93048bae38d9edba0e3a610a2f57cb955777ede99e06 WatchSource:0}: Error finding container 048b42f4d9fec22e382f93048bae38d9edba0e3a610a2f57cb955777ede99e06: Status 404 returned error can't find the container with id 048b42f4d9fec22e382f93048bae38d9edba0e3a610a2f57cb955777ede99e06 Jul 10 00:15:43.552530 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:43.552514 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc365a80243b37665c2c8c1b5f01aa50a.slice/crio-0cff6c5980bf0676b736e61002ce73e64ba66b7d04c963daf90afa7f05bf9ce9 WatchSource:0}: Error finding container 0cff6c5980bf0676b736e61002ce73e64ba66b7d04c963daf90afa7f05bf9ce9: Status 404 returned error can't find the container with id 0cff6c5980bf0676b736e61002ce73e64ba66b7d04c963daf90afa7f05bf9ce9 Jul 10 00:15:43.559732 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.559702 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:15:43.811523 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.811410 2548 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:15:43.984436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.984398 2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 00:15:43.984937 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.984406 2548 apiserver.go:52] "Watching apiserver" Jul 10 00:15:43.994871 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.994841 2548 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 00:15:43.996172 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.996147 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-gkxgp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal","openshift-multus/network-metrics-daemon-vgb4g","openshift-ovn-kubernetes/ovnkube-node-5lt5h","kube-system/global-pull-secret-syncer-mf9qj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg","openshift-cluster-node-tuning-operator/tuned-tjq2z","openshift-dns/node-resolver-9qxd8","openshift-multus/multus-8c2g2","openshift-multus/multus-additional-cni-plugins-8md9r","openshift-network-diagnostics/network-check-target-g6skz","openshift-network-operator/iptables-alerter-qnz5p","kube-system/konnectivity-agent-h4dsd","kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal"] Jul 10 00:15:43.997819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.997798 2548 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:15:43.998732 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.998710 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:43.999976 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:43.999957 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.001076 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.001057 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.001175 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.001133 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:44.001564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.001547 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.001564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.001575 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.001710 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.001589 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pk5p7\"" Jul 10 00:15:44.002451 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.002435 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.003018 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.002997 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 00:15:44.003117 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.003066 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 00:15:44.003117 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.003010 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c29g4\"" Jul 10 00:15:44.003939 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.003922 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.004039 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.004004 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:44.005079 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.005062 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 00:15:44.005533 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.005513 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.006200 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.006178 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.006616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.006389 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 00:15:44.006616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.006457 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 00:15:44.006616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.006507 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9lc9q\"" Jul 10 00:15:44.006927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.006912 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.007865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.007840 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.008140 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.008278 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009208 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009443 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009683 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009695 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009848 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.010067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.009992 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-79x2w\"" Jul 10 00:15:44.010764 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.010538 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 00:15:44.010764 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.010641 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wf5ff\"" Jul 10 00:15:44.011425 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.011405 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.014045 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.014021 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 00:15:44.014134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.014069 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.014134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.014123 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qs4pw\"" Jul 10 00:15:44.014232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.014155 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.014561 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.014548 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.015943 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.015920 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.017379 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.017358 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.017513 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.017391 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tp8qp\"" Jul 10 00:15:44.017513 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.017400 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.017513 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.017469 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:44.017652 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.017543 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:44.019010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.018989 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.019834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.019815 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 00:15:44.019940 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.019886 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 00:15:44.019940 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.019895 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t5l8d\"" Jul 10 00:15:44.021256 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.021239 2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xgf76" Jul 10 00:15:44.021608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.021553 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qsvl8\"" Jul 10 00:15:44.021608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.021559 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 00:15:44.021735 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.021657 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:15:44.021889 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.021876 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 00:15:44.029663 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.029637 2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xgf76" Jul 10 00:15:44.060758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.060737 2548 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 00:15:44.076863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076810 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cni-binary-copy\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.076863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076835 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-bin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.076863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076860 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6h97\" (UniqueName: \"kubernetes.io/projected/a87e7a17-472d-4f06-a5b9-96bbafb08123-kube-api-access-f6h97\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.077023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076879 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-system-cni-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.077023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076897 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-binary-copy\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.077023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076943 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-env-overrides\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.076975 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.077148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077020 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-systemd-units\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077045 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-script-lib\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077082 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzd57\" (UniqueName: \"kubernetes.io/projected/3732cd5e-42d6-4517-96a7-a6132fd22340-kube-api-access-jzd57\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077103 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9015bc2-ec2b-4800-b911-1e190522f37e-hosts-file\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.077148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077131 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxzf\" (UniqueName: \"kubernetes.io/projected/f80f9e53-8206-4724-9677-d1c46b2887f4-kube-api-access-zhxzf\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.077294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077155 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-cnibin\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.077294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077180 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-kube-api-access-zr4lj\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.077294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077205 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9015bc2-ec2b-4800-b911-1e190522f37e-tmp-dir\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.077294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077236 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxn2\" (UniqueName: \"kubernetes.io/projected/e9015bc2-ec2b-4800-b911-1e190522f37e-kube-api-access-pgxn2\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.077294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077261 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-systemd\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.077436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077287 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077317 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-var-lib-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077351 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-netd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077394 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-multus\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.077436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077424 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-etc-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077484 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3732cd5e-42d6-4517-96a7-a6132fd22340-ovn-node-metrics-cert\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077508 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-socket-dir-parent\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.077619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077554 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-kubelet\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.077619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077581 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-multus-certs\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077623 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077652 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpr5w\" (UniqueName: \"kubernetes.io/projected/68f27071-8614-4c45-8f17-ce4587ecd3d2-kube-api-access-vpr5w\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077677 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-systemd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077699 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-config\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077730 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-modprobe-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.077776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077755 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmtg\" (UniqueName: \"kubernetes.io/projected/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-kube-api-access-4dmtg\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.078026 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077785 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-device-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.078026 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077828 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-sys-fs\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.078225 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.077866 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfb94\" (UniqueName: \"kubernetes.io/projected/482f31e5-175d-4782-8d99-e51748a8b929-kube-api-access-zfb94\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.078359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078251 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-konnectivity-ca\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.078359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078294 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-slash\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.078359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078330 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-conf-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.078587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078362 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-etc-kubernetes\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.078643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078611 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-dbus\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.078706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078685 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.078758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078742 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f33960f-5d2f-48a6-b429-d02cc187a29d-iptables-alerter-script\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.078810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078769 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-agent-certs\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.078863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078815 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-ovn\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.078863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078846 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-log-socket\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.078949 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078875 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-netns\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.078949 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078916 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-daemon-config\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.079046 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078948 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-kubernetes\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079046 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.078995 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.079046 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079027 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:44.079189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079073 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.079189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079122 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-os-release\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.079189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079178 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-hostroot\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.079323 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079208 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-tuned\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079323 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079253 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-netns\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.079323 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079282 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-node-log\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.079458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079330 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cnibin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.079458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079355 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80f9e53-8206-4724-9677-d1c46b2887f4-serviceca\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.079458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079416 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-conf\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079448 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-kubelet-config\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.079677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079513 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.079677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079545 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-socket-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.079677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079570 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.079677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079618 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80f9e53-8206-4724-9677-d1c46b2887f4-host\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.079677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079649 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079696 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079727 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-lib-modules\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079765 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-var-lib-kubelet\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079796 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-registration-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079828 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.079903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079872 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-kubelet\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079897 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-k8s-cni-cncf-io\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079960 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysconfig\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.079994 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-run\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080040 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-host\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080071 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwvf\" (UniqueName: \"kubernetes.io/projected/9f33960f-5d2f-48a6-b429-d02cc187a29d-kube-api-access-jpwvf\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080121 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-system-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080151 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-sys\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080189 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-tmp\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.080244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080222 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-os-release\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.080685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080267 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f33960f-5d2f-48a6-b429-d02cc187a29d-host-slash\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.080685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080302 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.080685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.080348 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-bin\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.157424 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.157351 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" event={"ID":"c365a80243b37665c2c8c1b5f01aa50a","Type":"ContainerStarted","Data":"0cff6c5980bf0676b736e61002ce73e64ba66b7d04c963daf90afa7f05bf9ce9"} Jul 10 00:15:44.159108 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.159069 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"47aa25279bcd0f53cb13fb27c7635d7d","Type":"ContainerStarted","Data":"048b42f4d9fec22e382f93048bae38d9edba0e3a610a2f57cb955777ede99e06"} Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181467 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-netns\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181525 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-node-log\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181556 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cnibin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181581 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80f9e53-8206-4724-9677-d1c46b2887f4-serviceca\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181603 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-node-log\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181607 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-conf\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181655 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-kubelet-config\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181678 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181702 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-socket-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181711 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cnibin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181726 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181746 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-conf\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181767 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80f9e53-8206-4724-9677-d1c46b2887f4-host\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181797 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181826 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-etc-selinux\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181827 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181867 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-lib-modules\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.184596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181872 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181893 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-var-lib-kubelet\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181914 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80f9e53-8206-4724-9677-d1c46b2887f4-host\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181928 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-registration-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181959 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.181986 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-kubelet\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182014 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-k8s-cni-cncf-io\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182044 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysconfig\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182056 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysctl-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182070 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-run\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182097 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-host\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182123 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwvf\" (UniqueName: \"kubernetes.io/projected/9f33960f-5d2f-48a6-b429-d02cc187a29d-kube-api-access-jpwvf\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182150 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-system-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182171 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-registration-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182186 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-sys\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182210 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-tmp\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182239 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-kubelet-config\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.185643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182242 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-os-release\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182280 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f33960f-5d2f-48a6-b429-d02cc187a29d-host-slash\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182306 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182316 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80f9e53-8206-4724-9677-d1c46b2887f4-serviceca\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182336 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-bin\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182365 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cni-binary-copy\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182394 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-bin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182421 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6h97\" (UniqueName: \"kubernetes.io/projected/a87e7a17-472d-4f06-a5b9-96bbafb08123-kube-api-access-f6h97\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182450 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-system-cni-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182476 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-binary-copy\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182502 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-env-overrides\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182529 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182556 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-systemd-units\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182602 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-script-lib\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182628 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzd57\" (UniqueName: \"kubernetes.io/projected/3732cd5e-42d6-4517-96a7-a6132fd22340-kube-api-access-jzd57\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182656 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9015bc2-ec2b-4800-b911-1e190522f37e-hosts-file\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182686 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxzf\" (UniqueName: \"kubernetes.io/projected/f80f9e53-8206-4724-9677-d1c46b2887f4-kube-api-access-zhxzf\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.186589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182715 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-cnibin\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182741 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-kube-api-access-zr4lj\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182766 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9015bc2-ec2b-4800-b911-1e190522f37e-tmp-dir\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182793 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxn2\" (UniqueName: \"kubernetes.io/projected/e9015bc2-ec2b-4800-b911-1e190522f37e-kube-api-access-pgxn2\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182818 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-systemd\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182846 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182879 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-var-lib-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182911 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-netd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182940 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-multus\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182971 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-etc-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182997 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3732cd5e-42d6-4517-96a7-a6132fd22340-ovn-node-metrics-cert\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183004 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-sysconfig\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183023 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-system-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183059 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-socket-dir-parent\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183074 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-k8s-cni-cncf-io\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183083 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-kubelet\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183102 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-multus-certs\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.187465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183121 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183129 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-sys\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183153 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpr5w\" (UniqueName: \"kubernetes.io/projected/68f27071-8614-4c45-8f17-ce4587ecd3d2-kube-api-access-vpr5w\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183197 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-systemd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183224 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-config\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183250 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-modprobe-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183276 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmtg\" (UniqueName: \"kubernetes.io/projected/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-kube-api-access-4dmtg\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183305 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-device-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183334 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-sys-fs\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183361 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfb94\" (UniqueName: \"kubernetes.io/projected/482f31e5-175d-4782-8d99-e51748a8b929-kube-api-access-zfb94\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183389 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-konnectivity-ca\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183382 2548 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183417 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-slash\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183443 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-conf-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183469 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-etc-kubernetes\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183497 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-dbus\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183524 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.188397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183574 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f33960f-5d2f-48a6-b429-d02cc187a29d-iptables-alerter-script\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183601 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-agent-certs\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183628 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-ovn\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183652 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-socket-dir-parent\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183748 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-run\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183773 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183827 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-kubelet\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183867 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-multus-certs\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183966 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-systemd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.183656 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-log-socket\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184024 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-netns\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184055 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-daemon-config\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184112 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-kubernetes\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184143 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184190 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184219 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184248 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-os-release\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184279 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-hostroot\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.189313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184305 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-tuned\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184447 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-script-lib\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184511 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-ovnkube-config\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184509 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184635 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-modprobe-d\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184641 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-socket-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184698 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-var-lib-kubelet\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184735 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f33960f-5d2f-48a6-b429-d02cc187a29d-host-slash\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184774 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182308 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-os-release\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184822 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-bin\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.184879 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-sys-fs\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185155 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9015bc2-ec2b-4800-b911-1e190522f37e-hosts-file\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185338 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-cni-binary-copy\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185365 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-cnibin\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185408 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a87e7a17-472d-4f06-a5b9-96bbafb08123-device-dir\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185461 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-bin\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185517 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-konnectivity-ca\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.190254 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185599 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-slash\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185645 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-conf-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185684 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-etc-kubernetes\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185705 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-system-cni-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185790 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4921e36d-183c-4039-a49d-56ede70a834c-dbus\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185850 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9015bc2-ec2b-4800-b911-1e190522f37e-tmp-dir\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185864 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-run-netns\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.185935 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/482f31e5-175d-4782-8d99-e51748a8b929-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.185963 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.186066 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:15:44.686030931 +0000 UTC m=+3.165918483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186131 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-systemd\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186187 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186235 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-var-lib-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.182880 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186284 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-run-netns\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186304 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-kubelet\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186365 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-host-cni-netd\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.191206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186411 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-host\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186454 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-host-var-lib-cni-multus\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186444 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9f33960f-5d2f-48a6-b429-d02cc187a29d-iptables-alerter-script\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186519 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-etc-openvswitch\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.186618 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.186670 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:44.686650521 +0000 UTC m=+3.166538064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186818 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/482f31e5-175d-4782-8d99-e51748a8b929-cni-binary-copy\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186844 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-kubernetes\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186889 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-systemd-units\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186904 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-log-socket\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186942 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3732cd5e-42d6-4517-96a7-a6132fd22340-run-ovn\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.186984 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-cni-dir\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.187017 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-os-release\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.187044 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3732cd5e-42d6-4517-96a7-a6132fd22340-env-overrides\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.187117 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-lib-modules\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.187150 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-hostroot\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.187515 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-multus-daemon-config\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.190189 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-etc-tuned\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.192150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.190631 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4-agent-certs\") pod \"konnectivity-agent-h4dsd\" (UID: \"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4\") " pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.193010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.190743 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3732cd5e-42d6-4517-96a7-a6132fd22340-ovn-node-metrics-cert\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.193010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.192727 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-tmp\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.202472 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.202447 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:44.202623 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.202480 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:44.202623 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.202496 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:44.202623 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.202582 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:44.702546039 +0000 UTC m=+3.182433574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:44.205187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.205158 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzd57\" (UniqueName: \"kubernetes.io/projected/3732cd5e-42d6-4517-96a7-a6132fd22340-kube-api-access-jzd57\") pod \"ovnkube-node-5lt5h\" (UID: \"3732cd5e-42d6-4517-96a7-a6132fd22340\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.205771 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxn2\" (UniqueName: \"kubernetes.io/projected/e9015bc2-ec2b-4800-b911-1e190522f37e-kube-api-access-pgxn2\") pod \"node-resolver-9qxd8\" (UID: \"e9015bc2-ec2b-4800-b911-1e190522f37e\") " pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.206215 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6h97\" (UniqueName: \"kubernetes.io/projected/a87e7a17-472d-4f06-a5b9-96bbafb08123-kube-api-access-f6h97\") pod \"aws-ebs-csi-driver-node-h5llg\" (UID: \"a87e7a17-472d-4f06-a5b9-96bbafb08123\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.206552 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwvf\" (UniqueName: \"kubernetes.io/projected/9f33960f-5d2f-48a6-b429-d02cc187a29d-kube-api-access-jpwvf\") pod \"iptables-alerter-qnz5p\" (UID: \"9f33960f-5d2f-48a6-b429-d02cc187a29d\") " pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.206664 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4lj\" (UniqueName: \"kubernetes.io/projected/3dcab713-d3de-40c9-9c69-bb98ce4d93f0-kube-api-access-zr4lj\") pod \"multus-8c2g2\" (UID: \"3dcab713-d3de-40c9-9c69-bb98ce4d93f0\") " pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.207160 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmtg\" (UniqueName: \"kubernetes.io/projected/0ad77260-9bf7-4f88-81c4-1bbabcbb0605-kube-api-access-4dmtg\") pod \"tuned-tjq2z\" (UID: \"0ad77260-9bf7-4f88-81c4-1bbabcbb0605\") " pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.207244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.207174 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpr5w\" (UniqueName: \"kubernetes.io/projected/68f27071-8614-4c45-8f17-ce4587ecd3d2-kube-api-access-vpr5w\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.207623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.207373 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxzf\" (UniqueName: \"kubernetes.io/projected/f80f9e53-8206-4724-9677-d1c46b2887f4-kube-api-access-zhxzf\") pod \"node-ca-gkxgp\" (UID: \"f80f9e53-8206-4724-9677-d1c46b2887f4\") " pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.209457 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.209432 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfb94\" (UniqueName: \"kubernetes.io/projected/482f31e5-175d-4782-8d99-e51748a8b929-kube-api-access-zfb94\") pod \"multus-additional-cni-plugins-8md9r\" (UID: \"482f31e5-175d-4782-8d99-e51748a8b929\") " pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.310475 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.310443 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" Jul 10 00:15:44.316950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.316928 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:15:44.325352 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.325323 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:15:44.326155 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.326123 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad77260_9bf7_4f88_81c4_1bbabcbb0605.slice/crio-1caff0dc1419e3d3bd16a45f78fe884c3719f19f077265083183ea76b8fb355b WatchSource:0}: Error finding container 1caff0dc1419e3d3bd16a45f78fe884c3719f19f077265083183ea76b8fb355b: Status 404 returned error can't find the container with id 1caff0dc1419e3d3bd16a45f78fe884c3719f19f077265083183ea76b8fb355b Jul 10 00:15:44.328523 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.328502 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" Jul 10 00:15:44.334120 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.334080 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8c2g2" Jul 10 00:15:44.339329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.339156 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkxgp" Jul 10 00:15:44.340039 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.340010 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4cf4c8_026c_40d4_b0ab_18b4f6e404b4.slice/crio-990b182a2555867d6e69681f460e6218dd13cdf2dd5eac7b507cc83a23fbc05b WatchSource:0}: Error finding container 990b182a2555867d6e69681f460e6218dd13cdf2dd5eac7b507cc83a23fbc05b: Status 404 returned error can't find the container with id 990b182a2555867d6e69681f460e6218dd13cdf2dd5eac7b507cc83a23fbc05b Jul 10 00:15:44.345535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.345326 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9qxd8" Jul 10 00:15:44.350545 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.350380 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8md9r" Jul 10 00:15:44.351481 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.351448 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87e7a17_472d_4f06_a5b9_96bbafb08123.slice/crio-cf91cc32712b4dd618de4d47da444db31211f40ff86c51572f2e5a861fb9dac7 WatchSource:0}: Error finding container cf91cc32712b4dd618de4d47da444db31211f40ff86c51572f2e5a861fb9dac7: Status 404 returned error can't find the container with id cf91cc32712b4dd618de4d47da444db31211f40ff86c51572f2e5a861fb9dac7 Jul 10 00:15:44.354335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.354127 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qnz5p" Jul 10 00:15:44.358933 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.358895 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcab713_d3de_40c9_9c69_bb98ce4d93f0.slice/crio-cfd647085d32f5727246d0b3e3ac1d01c723b7af618c86ac9b1632e7ce14b4d2 WatchSource:0}: Error finding container cfd647085d32f5727246d0b3e3ac1d01c723b7af618c86ac9b1632e7ce14b4d2: Status 404 returned error can't find the container with id cfd647085d32f5727246d0b3e3ac1d01c723b7af618c86ac9b1632e7ce14b4d2 Jul 10 00:15:44.361709 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.361677 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80f9e53_8206_4724_9677_d1c46b2887f4.slice/crio-933183e92501fb835875a62e8d57f89c4c1f32459c3ee1f2f80512456284270b WatchSource:0}: Error finding container 933183e92501fb835875a62e8d57f89c4c1f32459c3ee1f2f80512456284270b: Status 404 returned error can't find the container with id 933183e92501fb835875a62e8d57f89c4c1f32459c3ee1f2f80512456284270b Jul 10 00:15:44.372526 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.372502 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9015bc2_ec2b_4800_b911_1e190522f37e.slice/crio-13299ab09ac7b86f6e9467110e053822c49e7c756cd1be71726b900828802b54 WatchSource:0}: Error finding container 13299ab09ac7b86f6e9467110e053822c49e7c756cd1be71726b900828802b54: Status 404 returned error can't find the container with id 13299ab09ac7b86f6e9467110e053822c49e7c756cd1be71726b900828802b54 Jul 10 00:15:44.380718 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.380685 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482f31e5_175d_4782_8d99_e51748a8b929.slice/crio-e976fa0ff1283d4d9132165e1746b4b466a9e159e88b427915603b22ce152b36 WatchSource:0}: Error finding container e976fa0ff1283d4d9132165e1746b4b466a9e159e88b427915603b22ce152b36: Status 404 returned error can't find the container with id e976fa0ff1283d4d9132165e1746b4b466a9e159e88b427915603b22ce152b36 Jul 10 00:15:44.382460 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:15:44.382428 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f33960f_5d2f_48a6_b429_d02cc187a29d.slice/crio-537bf226d300e3bfcc03e833ae3cdb7198662952409268b3d68e746247fb73bb WatchSource:0}: Error finding container 537bf226d300e3bfcc03e833ae3cdb7198662952409268b3d68e746247fb73bb: Status 404 returned error can't find the container with id 537bf226d300e3bfcc03e833ae3cdb7198662952409268b3d68e746247fb73bb Jul 10 00:15:44.689096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.689014 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:44.689096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.689071 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:44.689274 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.689184 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:44.689274 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.689205 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:44.689274 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.689240 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:15:45.689226604 +0000 UTC m=+4.169114133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:44.689375 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.689277 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:45.689255131 +0000 UTC m=+4.169142662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:44.789435 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:44.789396 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:44.789633 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.789598 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:44.789633 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.789616 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:44.789633 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.789626 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:44.789756 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:44.789683 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:45.789666854 +0000 UTC m=+4.269554386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:45.032396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.032330 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:10:44 +0000 UTC" deadline="2027-12-22 07:59:43.607536044 +0000 UTC" Jul 10 00:15:45.032396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.032389 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12727h43m58.575150057s" Jul 10 00:15:45.161883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.161827 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4dsd" event={"ID":"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4","Type":"ContainerStarted","Data":"990b182a2555867d6e69681f460e6218dd13cdf2dd5eac7b507cc83a23fbc05b"} Jul 10 00:15:45.163159 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.163126 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"9398d76c224bffcadbb926aaf1ad20ccf004691af8f677eb632d818074becec7"} Jul 10 00:15:45.164409 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.164380 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnz5p" event={"ID":"9f33960f-5d2f-48a6-b429-d02cc187a29d","Type":"ContainerStarted","Data":"537bf226d300e3bfcc03e833ae3cdb7198662952409268b3d68e746247fb73bb"} Jul 10 00:15:45.165663 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.165636 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerStarted","Data":"e976fa0ff1283d4d9132165e1746b4b466a9e159e88b427915603b22ce152b36"} Jul 10 00:15:45.166784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.166757 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" event={"ID":"a87e7a17-472d-4f06-a5b9-96bbafb08123","Type":"ContainerStarted","Data":"cf91cc32712b4dd618de4d47da444db31211f40ff86c51572f2e5a861fb9dac7"} Jul 10 00:15:45.167931 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.167907 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" event={"ID":"0ad77260-9bf7-4f88-81c4-1bbabcbb0605","Type":"ContainerStarted","Data":"1caff0dc1419e3d3bd16a45f78fe884c3719f19f077265083183ea76b8fb355b"} Jul 10 00:15:45.169175 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.169149 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9qxd8" event={"ID":"e9015bc2-ec2b-4800-b911-1e190522f37e","Type":"ContainerStarted","Data":"13299ab09ac7b86f6e9467110e053822c49e7c756cd1be71726b900828802b54"} Jul 10 00:15:45.170441 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.170419 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkxgp" event={"ID":"f80f9e53-8206-4724-9677-d1c46b2887f4","Type":"ContainerStarted","Data":"933183e92501fb835875a62e8d57f89c4c1f32459c3ee1f2f80512456284270b"} Jul 10 00:15:45.171698 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.171674 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8c2g2" event={"ID":"3dcab713-d3de-40c9-9c69-bb98ce4d93f0","Type":"ContainerStarted","Data":"cfd647085d32f5727246d0b3e3ac1d01c723b7af618c86ac9b1632e7ce14b4d2"} Jul 10 00:15:45.695428 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.695387 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:45.695705 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.695479 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:45.695705 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.695643 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:45.695848 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.695712 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:47.695691934 +0000 UTC m=+6.175579465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:45.695848 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.695783 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:45.695848 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.695817 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:15:47.695805797 +0000 UTC m=+6.175693339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:45.796206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:45.796167 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:45.796369 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.796330 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:45.796369 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.796352 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:45.796369 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.796364 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:45.796541 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:45.796428 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:47.796408283 +0000 UTC m=+6.276295812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:46.032917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:46.032840 2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:10:44 +0000 UTC" deadline="2027-12-05 13:09:11.554869386 +0000 UTC" Jul 10 00:15:46.032917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:46.032883 2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12324h53m25.521990637s" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:46.153511 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:46.153668 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:46.154106 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:46.154209 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:46.155325 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:46.155807 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:46.155470 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:47.182128 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.181740 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" event={"ID":"c365a80243b37665c2c8c1b5f01aa50a","Type":"ContainerStarted","Data":"63d42f9aa192ce898089d2bcd68b7b1661f0263452f47f8bc5381dd66fc0b0a0"} Jul 10 00:15:47.184726 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.184691 2548 generic.go:356] "Generic (PLEG): container finished" podID="47aa25279bcd0f53cb13fb27c7635d7d" containerID="263f842262cda02bd888b8e32486f5f76f2b9ae6164048074741c8137de911b4" exitCode=0 Jul 10 00:15:47.184865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.184741 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"47aa25279bcd0f53cb13fb27c7635d7d","Type":"ContainerDied","Data":"263f842262cda02bd888b8e32486f5f76f2b9ae6164048074741c8137de911b4"} Jul 10 00:15:47.196777 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.196673 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" podStartSLOduration=4.196651435 podStartE2EDuration="4.196651435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:47.196158974 +0000 UTC m=+5.676046526" watchObservedRunningTime="2026-07-10 00:15:47.196651435 +0000 UTC m=+5.676538989" Jul 10 00:15:47.712103 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.712050 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:47.712307 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.712132 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:47.712307 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.712255 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:47.712418 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.712317 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.712297709 +0000 UTC m=+10.192185249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:47.712872 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.712734 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:47.712872 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.712803 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.71278105 +0000 UTC m=+10.192668633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:47.813731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:47.813039 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:47.813731 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.813235 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:47.813731 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.813255 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:47.813731 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.813268 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:47.813731 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:47.813332 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:51.813307971 +0000 UTC m=+10.293195558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:48.151755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:48.151620 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:48.151755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:48.151656 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:48.151755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:48.151622 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:48.152018 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:48.151769 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:48.152338 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:48.152198 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:48.152338 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:48.152294 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:50.154802 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:50.154925 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:50.154930 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:50.155047 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:50.155112 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:50.155267 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:50.155189 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:51.745092 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:51.745173 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.745307 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.745366 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:15:59.745348071 +0000 UTC m=+18.225235605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.745429 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:51.745485 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.745463 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:59.745450655 +0000 UTC m=+18.225338196 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:51.846646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:51.846532 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:51.846797 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.846727 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:51.846797 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.846752 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:51.846797 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.846766 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:51.846954 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:51.846835 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:15:59.846816365 +0000 UTC m=+18.326703905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:52.152816 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:52.152343 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:52.152816 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:52.152442 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:52.152816 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:52.152669 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:52.152816 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:52.152800 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:52.153147 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:52.152864 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:52.153147 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:52.152880 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:54.151677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:54.151459 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:54.151677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:54.151459 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:54.151677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:54.151621 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:54.152561 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:54.152485 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:54.152708 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:54.152648 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:54.152708 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:54.151617 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:56.151392 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:56.151340 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:56.151945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:56.151342 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:56.151945 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:56.151508 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:56.151945 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:56.151606 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:56.151945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:56.151350 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:56.151945 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:56.151696 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:56.204953 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:56.204910 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"47aa25279bcd0f53cb13fb27c7635d7d","Type":"ContainerStarted","Data":"ef94c2ebefe3932b11b27486df9cc2749090c76e596e9e8db7550b6a815f693f"} Jul 10 00:15:56.219332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:56.219278 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" podStartSLOduration=13.21926041 podStartE2EDuration="13.21926041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:15:56.218920455 +0000 UTC m=+14.698808003" watchObservedRunningTime="2026-07-10 00:15:56.21926041 +0000 UTC m=+14.699147957" Jul 10 00:15:58.151875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:58.151838 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:58.152363 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:58.151838 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:58.152363 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:58.151971 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:58.152363 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:58.151978 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:15:58.152363 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:58.152075 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:15:58.152363 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:58.152158 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:15:59.809913 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:59.809874 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:15:59.810394 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:59.809962 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:15:59.810394 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.809988 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:59.810394 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.810060 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:15.810043527 +0000 UTC m=+34.289931058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:15:59.810394 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.810081 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:59.810394 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.810176 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:16:15.810155924 +0000 UTC m=+34.290043460 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:15:59.910957 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:15:59.910904 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:15:59.911121 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.911098 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:15:59.911186 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.911126 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:15:59.911186 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.911139 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:15:59.911289 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:15:59.911194 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:15.911176493 +0000 UTC m=+34.391064041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:16:00.151569 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.151546 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:00.151671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.151572 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:00.151671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.151557 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:00.151671 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:00.151658 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:00.151838 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:00.151761 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:00.151887 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:00.151832 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:00.212834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.212807 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9qxd8" event={"ID":"e9015bc2-ec2b-4800-b911-1e190522f37e","Type":"ContainerStarted","Data":"0f7ba12d6714c53c109811b044d107c28a56f411777e48fb658d350859d745b8"} Jul 10 00:16:00.214126 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.214097 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkxgp" event={"ID":"f80f9e53-8206-4724-9677-d1c46b2887f4","Type":"ContainerStarted","Data":"89c25d91f19f61c58ed926751d668c3c65d1f40fa1f26a12c2008e2b6c6c2580"} Jul 10 00:16:00.215320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.215299 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h4dsd" event={"ID":"ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4","Type":"ContainerStarted","Data":"7d3110f8ef526e4a8349082a4f7c9bb955b35682537b4d2337a97085b2233bd6"} Jul 10 00:16:00.216650 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.216627 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerStarted","Data":"c63e02d7ae01cc6803554ce05566f1ab806ec3e22c65abc89324d34497a79ece"} Jul 10 00:16:00.217913 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.217893 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" event={"ID":"a87e7a17-472d-4f06-a5b9-96bbafb08123","Type":"ContainerStarted","Data":"a7898e25c72defb4fd6f07b2d96c26e5a68506a7fb15afa9618304af6179caf9"} Jul 10 00:16:00.219160 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.219132 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" event={"ID":"0ad77260-9bf7-4f88-81c4-1bbabcbb0605","Type":"ContainerStarted","Data":"b07b6cea3db0ca85e38baa12455b1c5dae417db13b0db4d76d6a3075b4fa82c5"} Jul 10 00:16:00.228066 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.228025 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9qxd8" podStartSLOduration=7.247583886 podStartE2EDuration="18.228012491s" totalImagesPullingTime="10.980428605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:00.227343908 +0000 UTC m=+18.707231456" watchObservedRunningTime="2026-07-10 00:16:00.228012491 +0000 UTC m=+18.707900045" Jul 10 00:16:00.255152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.255106 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h4dsd" podStartSLOduration=7.310131291 podStartE2EDuration="18.255089314s" totalImagesPullingTime="10.944958023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:00.240924835 +0000 UTC m=+18.720812383" watchObservedRunningTime="2026-07-10 00:16:00.255089314 +0000 UTC m=+18.734976867" Jul 10 00:16:00.255259 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:00.255233 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gkxgp" podStartSLOduration=7.265255556 podStartE2EDuration="18.255223036s" totalImagesPullingTime="10.98996748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:00.254746555 +0000 UTC m=+18.734634102" watchObservedRunningTime="2026-07-10 00:16:00.255223036 +0000 UTC m=+18.735110586" Jul 10 00:16:02.152784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.152748 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:02.152784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.152775 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:02.153481 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:02.152856 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:02.153481 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:02.152891 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:02.153481 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.152938 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:02.153481 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:02.153020 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:02.224379 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.224343 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qnz5p" event={"ID":"9f33960f-5d2f-48a6-b429-d02cc187a29d","Type":"ContainerStarted","Data":"287a9bbc8c8d1845650502574ee5ac00acea11a78233ad074e837a27543e85b0"} Jul 10 00:16:02.239118 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.239064 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tjq2z" podStartSLOduration=4.54307235 podStartE2EDuration="20.239048823s" totalImagesPullingTime="15.695976473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:00.297133169 +0000 UTC m=+18.777020719" watchObservedRunningTime="2026-07-10 00:16:02.239048823 +0000 UTC m=+20.718936381" Jul 10 00:16:02.243023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:02.242960 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qnz5p" podStartSLOduration=9.272510117 podStartE2EDuration="20.24293867s" totalImagesPullingTime="10.970428553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:02.238839056 +0000 UTC m=+20.718726604" watchObservedRunningTime="2026-07-10 00:16:02.24293867 +0000 UTC m=+20.722826221" Jul 10 00:16:03.254539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:03.254509 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:16:03.255249 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:03.255228 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:16:04.156553 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.156253 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:04.156553 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.156309 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:04.156553 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.156314 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:04.156840 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:04.156585 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:04.156840 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:04.156701 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:04.156840 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:04.156759 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:04.230571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.230539 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="c63e02d7ae01cc6803554ce05566f1ab806ec3e22c65abc89324d34497a79ece" exitCode=0 Jul 10 00:16:04.230757 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.230623 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"c63e02d7ae01cc6803554ce05566f1ab806ec3e22c65abc89324d34497a79ece"} Jul 10 00:16:04.233278 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.233133 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8c2g2" event={"ID":"3dcab713-d3de-40c9-9c69-bb98ce4d93f0","Type":"ContainerStarted","Data":"6819879abf6d0d307b26f6af62cb058a2ec77f9b555451c1011f3576c4f9a053"} Jul 10 00:16:04.235454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.235436 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:16:04.236607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.236582 2548 generic.go:356] "Generic (PLEG): container finished" podID="3732cd5e-42d6-4517-96a7-a6132fd22340" containerID="f88ce9ede9cd6c0cbfca5622c8a7a7ed370a5b17fc8dd9ea82cfa2277e76ae24" exitCode=1 Jul 10 00:16:04.236680 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.236654 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"f6d54f4b43486b7b029248e153c887c4ff95bb9582659543565b998fa882e5d3"} Jul 10 00:16:04.236768 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.236734 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerDied","Data":"f88ce9ede9cd6c0cbfca5622c8a7a7ed370a5b17fc8dd9ea82cfa2277e76ae24"} Jul 10 00:16:04.236829 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.236793 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"0be3fd4c3299ff2b7b3d4c2f7d7e549b8f79ec850291a6de5e81c1d5b61ad99a"} Jul 10 00:16:04.274698 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:04.274637 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8c2g2" podStartSLOduration=2.648097362 podStartE2EDuration="22.274617652s" totalImagesPullingTime="19.62652029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:04.274371449 +0000 UTC m=+22.754258997" watchObservedRunningTime="2026-07-10 00:16:04.274617652 +0000 UTC m=+22.754505201" Jul 10 00:16:05.067031 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.066694 2548 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 00:16:05.240811 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.240773 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" event={"ID":"a87e7a17-472d-4f06-a5b9-96bbafb08123","Type":"ContainerStarted","Data":"23b01fd8f95d1417afe347285528414d8c1e78db87496aa7d06cce41b18a5f64"} Jul 10 00:16:05.243215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.243195 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:16:05.243630 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.243607 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"e12e9a11fa91455f605d4eb59ae23af250d2384a82743a09a01748bab8be28ad"} Jul 10 00:16:05.243686 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.243676 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"8a7737281fc0c98e31214b78407248ca5537e85d91c32eb83d2f736f82048dc4"} Jul 10 00:16:05.243726 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:05.243707 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"34cc9592dacdbc6e207d49f6c8b5fdbf674b48b16a0ee6620ecc1f5979b286e4"} Jul 10 00:16:06.042062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.041968 2548 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T00:16:05.066869994Z","UUID":"4c0b0318-c842-4b2d-8f1c-de47be1c9687","Handler":null,"Name":"","Endpoint":""} Jul 10 00:16:06.043521 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.043484 2548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 00:16:06.043657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.043533 2548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 00:16:06.152015 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.151978 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:06.152208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.151982 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:06.152208 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:06.152111 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:06.152208 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:06.152175 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:06.152208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:06.151998 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:06.152408 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:06.152267 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:07.252722 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:07.252676 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" event={"ID":"a87e7a17-472d-4f06-a5b9-96bbafb08123","Type":"ContainerStarted","Data":"25d2e56ca2d92e1835cfde97613c655fb0f17309c92be9e0012886486e72b8c5"} Jul 10 00:16:07.255546 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:07.255526 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:16:07.255867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:07.255844 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"43f2a8f7956572dc94093ba05190f991af8dc9b1499e36c2027cb9347220e792"} Jul 10 00:16:07.270637 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:07.270587 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h5llg" podStartSLOduration=3.392888804 podStartE2EDuration="25.270569993s" totalImagesPullingTime="21.877681189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:07.26875683 +0000 UTC m=+25.748644379" watchObservedRunningTime="2026-07-10 00:16:07.270569993 +0000 UTC m=+25.750457541" Jul 10 00:16:08.151462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:08.151431 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:08.151636 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:08.151430 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:08.151636 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:08.151558 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:08.151636 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:08.151606 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:08.151748 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:08.151704 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:08.151837 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:08.151803 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:10.152155 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.151947 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:10.152833 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.151954 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:10.152833 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:10.152275 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:10.152833 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.151996 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:10.152833 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:10.152373 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:10.152833 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:10.152451 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:10.266763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.266736 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:16:10.267199 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.267175 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"8304d9f0bc780a76ea47f9abc83ff3f231e076424e2ddf028ec4c499405eb3ad"} Jul 10 00:16:10.267521 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.267433 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:10.267657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.267623 2548 scope.go:117] "RemoveContainer" containerID="f88ce9ede9cd6c0cbfca5622c8a7a7ed370a5b17fc8dd9ea82cfa2277e76ae24" Jul 10 00:16:10.269383 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.269363 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerStarted","Data":"1aaf86d7bf27881520de51290139e222e89a645806c82985399446a236be6d43"} Jul 10 00:16:10.284383 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.284361 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:10.342336 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.342307 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:10.443828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.443792 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:16:10.443968 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.443913 2548 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 00:16:10.444488 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:10.444464 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h4dsd" Jul 10 00:16:11.275153 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.275124 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:16:11.275652 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.275552 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" event={"ID":"3732cd5e-42d6-4517-96a7-a6132fd22340","Type":"ContainerStarted","Data":"ce1cb95ef4c6f18f6238c85843944858b7cc25e4ef0fcb7db7b7d9d5f3cd7fc7"} Jul 10 00:16:11.275857 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.275834 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:11.277445 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.277424 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="1aaf86d7bf27881520de51290139e222e89a645806c82985399446a236be6d43" exitCode=0 Jul 10 00:16:11.277544 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.277486 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"1aaf86d7bf27881520de51290139e222e89a645806c82985399446a236be6d43"} Jul 10 00:16:11.292053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.292032 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:11.303906 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:11.303865 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" podStartSLOduration=9.695698364 podStartE2EDuration="29.303850506s" totalImagesPullingTime="19.608152142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:11.303411142 +0000 UTC m=+29.783298691" watchObservedRunningTime="2026-07-10 00:16:11.303850506 +0000 UTC m=+29.783738054" Jul 10 00:16:12.121963 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.121929 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mf9qj"] Jul 10 00:16:12.122144 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.122094 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:12.122231 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:12.122208 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:12.125146 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.125118 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6skz"] Jul 10 00:16:12.125347 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.125243 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:12.125440 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:12.125340 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:12.125875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.125831 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vgb4g"] Jul 10 00:16:12.125978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.125939 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:12.126040 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:12.126024 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:12.281773 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:12.281518 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerStarted","Data":"4b725e47526a921afb6a4f90537dea2b1e6c151f465e63ba5a43d4c298b4f39e"} Jul 10 00:16:13.285810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:13.285726 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="4b725e47526a921afb6a4f90537dea2b1e6c151f465e63ba5a43d4c298b4f39e" exitCode=0 Jul 10 00:16:13.285810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:13.285761 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"4b725e47526a921afb6a4f90537dea2b1e6c151f465e63ba5a43d4c298b4f39e"} Jul 10 00:16:13.655523 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:13.655477 2548 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4dsd_ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4/konnectivity-agent/0.log" limit=1024 Jul 10 00:16:13.758031 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:13.758004 2548 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h4dsd_ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4/konnectivity-agent/0.log" limit=1024 Jul 10 00:16:14.152309 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:14.152275 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:14.152457 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:14.152275 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:14.152457 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:14.152389 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:14.152549 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:14.152285 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:14.152549 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:14.152482 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:14.152615 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:14.152597 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:15.292757 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:15.292719 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="5a51cac74f910fa6a315e9ecfc3a4307ea002c77d38706ed52a7ae74087fb14a" exitCode=0 Jul 10 00:16:15.293520 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:15.292769 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"5a51cac74f910fa6a315e9ecfc3a4307ea002c77d38706ed52a7ae74087fb14a"} Jul 10 00:16:15.829438 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:15.829404 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:15.829606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:15.829451 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:15.829606 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.829574 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:16:15.829680 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.829631 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:47.829616552 +0000 UTC m=+66.309504081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:16:15.829680 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.829574 2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:16:15.829764 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.829720 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret podName:4921e36d-183c-4039-a49d-56ede70a834c nodeName:}" failed. No retries permitted until 2026-07-10 00:16:47.829693636 +0000 UTC m=+66.309581166 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret") pod "global-pull-secret-syncer-mf9qj" (UID: "4921e36d-183c-4039-a49d-56ede70a834c") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:16:15.929942 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:15.929910 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:15.930102 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.930044 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:16:15.930102 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.930061 2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:16:15.930102 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.930070 2548 projected.go:196] Error preparing data for projected volume kube-api-access-47qcr for pod openshift-network-diagnostics/network-check-target-g6skz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:16:15.930199 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:15.930125 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr podName:9023eeb1-3c76-469f-843b-118bb40ace13 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:47.930106335 +0000 UTC m=+66.409993878 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-47qcr" (UniqueName: "kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr") pod "network-check-target-g6skz" (UID: "9023eeb1-3c76-469f-843b-118bb40ace13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:16:16.152210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:16.152129 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:16.152210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:16.152155 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:16.152210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:16.152168 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:16.152410 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:16.152269 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:16:16.152410 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:16.152386 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mf9qj" podUID="4921e36d-183c-4039-a49d-56ede70a834c" Jul 10 00:16:16.152507 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:16.152473 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6skz" podUID="9023eeb1-3c76-469f-843b-118bb40ace13" Jul 10 00:16:17.796411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.796197 2548 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeReady" Jul 10 00:16:17.796773 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.796582 2548 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 00:16:17.836245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.836217 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z5hkd"] Jul 10 00:16:17.838357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.838334 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:17.839844 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.839820 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7ssjm"] Jul 10 00:16:17.841241 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.841217 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 00:16:17.841357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.841229 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crl8x\"" Jul 10 00:16:17.841357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.841272 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 00:16:17.841673 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.841650 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:17.844195 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.844176 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 00:16:17.844454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.844430 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 00:16:17.844570 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.844436 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vlbng\"" Jul 10 00:16:17.844570 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.844471 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 00:16:17.851165 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.851139 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5hkd"] Jul 10 00:16:17.870673 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.870642 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ssjm"] Jul 10 00:16:17.940554 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940521 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:17.940665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940585 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0559a2bd-a537-451d-ba80-a6a74fbf2067-config-volume\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:17.940665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940604 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0559a2bd-a537-451d-ba80-a6a74fbf2067-tmp-dir\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:17.940665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940620 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtjs\" (UniqueName: \"kubernetes.io/projected/0559a2bd-a537-451d-ba80-a6a74fbf2067-kube-api-access-6wtjs\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:17.940665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940653 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2l55\" (UniqueName: \"kubernetes.io/projected/5c7a8fbe-0641-4a83-91aa-28ce36b25468-kube-api-access-d2l55\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:17.940793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:17.940706 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c7a8fbe-0641-4a83-91aa-28ce36b25468-cert\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.041332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041301 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0559a2bd-a537-451d-ba80-a6a74fbf2067-config-volume\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.041462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041343 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0559a2bd-a537-451d-ba80-a6a74fbf2067-tmp-dir\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.041462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041362 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtjs\" (UniqueName: \"kubernetes.io/projected/0559a2bd-a537-451d-ba80-a6a74fbf2067-kube-api-access-6wtjs\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.041462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041379 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2l55\" (UniqueName: \"kubernetes.io/projected/5c7a8fbe-0641-4a83-91aa-28ce36b25468-kube-api-access-d2l55\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.041462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041442 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c7a8fbe-0641-4a83-91aa-28ce36b25468-cert\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.041682 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041521 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.041682 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:18.041661 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:18.041818 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:18.041721 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:18.541702705 +0000 UTC m=+37.021590230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:18.041818 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.041725 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0559a2bd-a537-451d-ba80-a6a74fbf2067-tmp-dir\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.042136 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.042119 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0559a2bd-a537-451d-ba80-a6a74fbf2067-config-volume\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.046077 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.046056 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c7a8fbe-0641-4a83-91aa-28ce36b25468-cert\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.050543 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.050450 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtjs\" (UniqueName: \"kubernetes.io/projected/0559a2bd-a537-451d-ba80-a6a74fbf2067-kube-api-access-6wtjs\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.050827 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.050804 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2l55\" (UniqueName: \"kubernetes.io/projected/5c7a8fbe-0641-4a83-91aa-28ce36b25468-kube-api-access-d2l55\") pod \"ingress-canary-7ssjm\" (UID: \"5c7a8fbe-0641-4a83-91aa-28ce36b25468\") " pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.151481 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.151444 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:18.151657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.151485 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:18.151657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.151447 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:18.154916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.154595 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:16:18.154916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.154643 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:16:18.154916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.154715 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qmtcf\"" Jul 10 00:16:18.154916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.154595 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:16:18.155140 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.155044 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:16:18.155140 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.155045 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qn6qf\"" Jul 10 00:16:18.158469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.158450 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7ssjm" Jul 10 00:16:18.302210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.302138 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7ssjm"] Jul 10 00:16:18.305969 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:16:18.305941 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7a8fbe_0641_4a83_91aa_28ce36b25468.slice/crio-59169a54de387bacd4f38a4ebe7e17bf74748876122faa0659218a20d7038396 WatchSource:0}: Error finding container 59169a54de387bacd4f38a4ebe7e17bf74748876122faa0659218a20d7038396: Status 404 returned error can't find the container with id 59169a54de387bacd4f38a4ebe7e17bf74748876122faa0659218a20d7038396 Jul 10 00:16:18.545680 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:18.545646 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:18.545832 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:18.545764 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:18.545832 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:18.545813 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:19.545799529 +0000 UTC m=+38.025687058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:19.303285 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:19.303228 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ssjm" event={"ID":"5c7a8fbe-0641-4a83-91aa-28ce36b25468","Type":"ContainerStarted","Data":"59169a54de387bacd4f38a4ebe7e17bf74748876122faa0659218a20d7038396"} Jul 10 00:16:19.554112 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:19.553937 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:19.554112 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:19.554072 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:19.554332 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:19.554147 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:21.554127016 +0000 UTC m=+40.034014554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:21.571185 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:21.571146 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:21.571658 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:21.571324 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:21.571658 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:21.571407 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:25.571386299 +0000 UTC m=+44.051273844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:22.313436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:22.313402 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="701e1bc4d80936c5af270c19eed6e11f3dcaca84ab9483acf71b63c84252d9bd" exitCode=0 Jul 10 00:16:22.313645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:22.313509 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"701e1bc4d80936c5af270c19eed6e11f3dcaca84ab9483acf71b63c84252d9bd"} Jul 10 00:16:22.315011 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:22.314993 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7ssjm" event={"ID":"5c7a8fbe-0641-4a83-91aa-28ce36b25468","Type":"ContainerStarted","Data":"de79b3ff8cb72b46c1a5edba7077bf992bd4da4c89314a99433db57b7e080343"} Jul 10 00:16:22.352072 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:22.352019 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7ssjm" podStartSLOduration=1.701890353 podStartE2EDuration="5.352002587s" totalImagesPullingTime="3.650112234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:22.351267001 +0000 UTC m=+40.831154549" watchObservedRunningTime="2026-07-10 00:16:22.352002587 +0000 UTC m=+40.831890183" Jul 10 00:16:23.319843 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:23.319807 2548 generic.go:356] "Generic (PLEG): container finished" podID="482f31e5-175d-4782-8d99-e51748a8b929" containerID="d02b55b892f4c6542d5e7d3d63d86468cc5c1d5c340593e28f45ab2ecdd773b5" exitCode=0 Jul 10 00:16:23.320224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:23.319893 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerDied","Data":"d02b55b892f4c6542d5e7d3d63d86468cc5c1d5c340593e28f45ab2ecdd773b5"} Jul 10 00:16:24.325537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:24.325488 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8md9r" event={"ID":"482f31e5-175d-4782-8d99-e51748a8b929","Type":"ContainerStarted","Data":"702c695080a8bacc618fef7cccc7d90a68841a29c81572bc0553d35a13a532a1"} Jul 10 00:16:24.348626 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:24.348563 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8md9r" podStartSLOduration=16.866733999 podStartE2EDuration="42.348543259s" totalImagesPullingTime="25.48180926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:24.347901663 +0000 UTC m=+42.827789221" watchObservedRunningTime="2026-07-10 00:16:24.348543259 +0000 UTC m=+42.828430809" Jul 10 00:16:25.602105 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:25.602069 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:25.602524 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:25.602222 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:25.602524 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:25.602285 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:33.602270002 +0000 UTC m=+52.082157528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:33.660576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:33.660538 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:33.661146 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:33.660705 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:33.661146 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:33.660791 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:16:49.660769554 +0000 UTC m=+68.140657094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:43.301457 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:43.301432 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lt5h" Jul 10 00:16:47.843574 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.843537 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:47.844067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.843616 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:16:47.846651 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.846625 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:16:47.846712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.846655 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:16:47.853931 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:47.853915 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:16:47.854008 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:47.853974 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:51.853959395 +0000 UTC m=+130.333846921 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : secret "metrics-daemon-secret" not found Jul 10 00:16:47.855977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.855959 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4921e36d-183c-4039-a49d-56ede70a834c-original-pull-secret\") pod \"global-pull-secret-syncer-mf9qj\" (UID: \"4921e36d-183c-4039-a49d-56ede70a834c\") " pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:47.864153 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.864131 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mf9qj" Jul 10 00:16:47.943899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.943870 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:47.946563 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.946545 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:16:47.957048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.957024 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:16:47.967228 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.967206 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qcr\" (UniqueName: \"kubernetes.io/projected/9023eeb1-3c76-469f-843b-118bb40ace13-kube-api-access-47qcr\") pod \"network-check-target-g6skz\" (UID: \"9023eeb1-3c76-469f-843b-118bb40ace13\") " pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:47.977237 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:47.977216 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mf9qj"] Jul 10 00:16:47.980462 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:16:47.980436 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4921e36d_183c_4039_a49d_56ede70a834c.slice/crio-d127036007c32ba6a5454fb8986ae1f816da1c96f21ed9f6d8656c3a07bcb893 WatchSource:0}: Error finding container d127036007c32ba6a5454fb8986ae1f816da1c96f21ed9f6d8656c3a07bcb893: Status 404 returned error can't find the container with id d127036007c32ba6a5454fb8986ae1f816da1c96f21ed9f6d8656c3a07bcb893 Jul 10 00:16:48.189136 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:48.189060 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qmtcf\"" Jul 10 00:16:48.197160 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:48.197139 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:48.313096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:48.313066 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6skz"] Jul 10 00:16:48.316016 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:16:48.315990 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9023eeb1_3c76_469f_843b_118bb40ace13.slice/crio-d74a72f2ae6be463f5fcceea6f666dda8861395bf4bae1efd0b76a98ce182641 WatchSource:0}: Error finding container d74a72f2ae6be463f5fcceea6f666dda8861395bf4bae1efd0b76a98ce182641: Status 404 returned error can't find the container with id d74a72f2ae6be463f5fcceea6f666dda8861395bf4bae1efd0b76a98ce182641 Jul 10 00:16:48.381812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:48.381772 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6skz" event={"ID":"9023eeb1-3c76-469f-843b-118bb40ace13","Type":"ContainerStarted","Data":"d74a72f2ae6be463f5fcceea6f666dda8861395bf4bae1efd0b76a98ce182641"} Jul 10 00:16:48.382775 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:48.382744 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mf9qj" event={"ID":"4921e36d-183c-4039-a49d-56ede70a834c","Type":"ContainerStarted","Data":"d127036007c32ba6a5454fb8986ae1f816da1c96f21ed9f6d8656c3a07bcb893"} Jul 10 00:16:49.753398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:49.753340 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:16:49.753946 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:49.753458 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:16:49.753946 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:16:49.753527 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:21.753512849 +0000 UTC m=+100.233400378 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:16:53.396307 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:53.396272 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mf9qj" event={"ID":"4921e36d-183c-4039-a49d-56ede70a834c","Type":"ContainerStarted","Data":"59ee68f673327bb7b22b79920eba0023774db9dcc8f7edf3db34aa54b22fb199"} Jul 10 00:16:53.397741 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:53.397716 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6skz" event={"ID":"9023eeb1-3c76-469f-843b-118bb40ace13","Type":"ContainerStarted","Data":"f554c66d628c585e854347b3df9268194604e276a66d1177a29eda2b14d10342"} Jul 10 00:16:53.397859 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:53.397848 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:16:53.411458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:53.411411 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mf9qj" podStartSLOduration=66.575082532 podStartE2EDuration="1m11.411398365s" totalImagesPullingTime="4.836315833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:53.410676939 +0000 UTC m=+71.890564487" watchObservedRunningTime="2026-07-10 00:16:53.411398365 +0000 UTC m=+71.891285912" Jul 10 00:16:53.426597 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:16:53.426545 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6skz" podStartSLOduration=66.928990488 podStartE2EDuration="1m11.426527435s" totalImagesPullingTime="4.497536947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:16:53.426203667 +0000 UTC m=+71.906091226" watchObservedRunningTime="2026-07-10 00:16:53.426527435 +0000 UTC m=+71.906414983" Jul 10 00:17:20.588504 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.588373 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv"] Jul 10 00:17:20.590790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.590771 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.593481 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.593450 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 00:17:20.593611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.593508 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 00:17:20.593611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.593570 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 00:17:20.593611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.593582 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 00:17:20.594930 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.594913 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 00:17:20.595001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.594956 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 00:17:20.595064 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.595040 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 00:17:20.603142 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.603122 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv"] Jul 10 00:17:20.660870 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.660827 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.660870 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.660870 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.661095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.660897 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.661095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.660914 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.661095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.660991 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.661095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.661024 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfr4h\" (UniqueName: \"kubernetes.io/projected/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-kube-api-access-nfr4h\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762136 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762174 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfr4h\" (UniqueName: \"kubernetes.io/projected/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-kube-api-access-nfr4h\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762214 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762238 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762317 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.762357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.762340 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.763095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.763069 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.764856 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.764834 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.764979 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.764875 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-ca\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.764979 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.764907 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.764979 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.764968 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.769679 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.769658 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfr4h\" (UniqueName: \"kubernetes.io/projected/d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92-kube-api-access-nfr4h\") pod \"cluster-proxy-proxy-agent-6f965c669c-ghgqv\" (UID: \"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:20.911743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:20.911694 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:17:21.030890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:21.030857 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv"] Jul 10 00:17:21.033970 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:17:21.033941 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4fd94ff_8d8c_4f49_bd5d_84d9daca9b92.slice/crio-db20f41a3655516c06ee59ac6dface02e98cdce6e92a6cf03e653afa4e48a160 WatchSource:0}: Error finding container db20f41a3655516c06ee59ac6dface02e98cdce6e92a6cf03e653afa4e48a160: Status 404 returned error can't find the container with id db20f41a3655516c06ee59ac6dface02e98cdce6e92a6cf03e653afa4e48a160 Jul 10 00:17:21.467180 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:21.467137 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerStarted","Data":"db20f41a3655516c06ee59ac6dface02e98cdce6e92a6cf03e653afa4e48a160"} Jul 10 00:17:21.770023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:21.769925 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:17:21.770404 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:17:21.770076 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:17:21.770404 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:17:21.770141 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:25.770126114 +0000 UTC m=+164.250013641 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:17:24.403609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:24.403576 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6skz" Jul 10 00:17:25.480001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:25.479964 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerStarted","Data":"84b8a8e615f8191d0eb35f3588d9402ca863f6b11a34760b938542e4b761ad30"} Jul 10 00:17:25.480375 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:25.480008 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerStarted","Data":"e671dbd207930fbf948793c53c74e27237079abf0234491b68f019a9ae85356e"} Jul 10 00:17:25.480375 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:25.480023 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerStarted","Data":"3e2441693f563f10cfea9b822607691873791562fdba3a642a77968abc5c18e4"} Jul 10 00:17:25.497070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:25.497022 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" podStartSLOduration=1.4477845839999999 podStartE2EDuration="5.497008591s" totalImagesPullingTime="4.049224007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:17:25.496480032 +0000 UTC m=+103.976367581" watchObservedRunningTime="2026-07-10 00:17:25.497008591 +0000 UTC m=+103.976896139" Jul 10 00:17:51.882215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:17:51.882179 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:17:51.882646 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:17:51.882299 2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:17:51.882646 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:17:51.882366 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs podName:68f27071-8614-4c45-8f17-ce4587ecd3d2 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:53.882349595 +0000 UTC m=+252.362237123 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs") pod "network-metrics-daemon-vgb4g" (UID: "68f27071-8614-4c45-8f17-ce4587ecd3d2") : secret "metrics-daemon-secret" not found Jul 10 00:18:20.851937 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:18:20.851888 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z5hkd" podUID="0559a2bd-a537-451d-ba80-a6a74fbf2067" Jul 10 00:18:21.180863 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:18:21.180768 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vgb4g" podUID="68f27071-8614-4c45-8f17-ce4587ecd3d2" Jul 10 00:18:21.635281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:18:21.635253 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:18:25.812774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:18:25.812713 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:18:25.813220 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:18:25.812868 2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:25.813220 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:18:25.812933 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls podName:0559a2bd-a537-451d-ba80-a6a74fbf2067 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:27.812915411 +0000 UTC m=+286.292802938 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls") pod "dns-default-z5hkd" (UID: "0559a2bd-a537-451d-ba80-a6a74fbf2067") : secret "dns-default-metrics-tls" not found Jul 10 00:18:33.152079 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:18:33.152031 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:19:36.677200 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.677161 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqf"] Jul 10 00:19:36.679123 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.679102 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.682977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.682953 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 00:19:36.683076 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.683032 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 00:19:36.684587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.684570 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 00:19:36.685083 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.685063 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2znzl\"" Jul 10 00:19:36.685479 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.685452 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 00:19:36.698532 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.698511 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqf"] Jul 10 00:19:36.791593 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.791559 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.791593 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.791595 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d21b32-6172-4399-889c-ea552b902408-data-volume\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.791825 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.791685 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.791825 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.791714 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvkc5\" (UniqueName: \"kubernetes.io/projected/f0d21b32-6172-4399-889c-ea552b902408-kube-api-access-kvkc5\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.791825 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.791733 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0d21b32-6172-4399-889c-ea552b902408-crio-socket\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.892794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892750 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.892794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892792 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d21b32-6172-4399-889c-ea552b902408-data-volume\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892832 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892856 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvkc5\" (UniqueName: \"kubernetes.io/projected/f0d21b32-6172-4399-889c-ea552b902408-kube-api-access-kvkc5\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892880 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0d21b32-6172-4399-889c-ea552b902408-crio-socket\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:36.892889 2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:36.892954 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.892962 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f0d21b32-6172-4399-889c-ea552b902408-crio-socket\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:36.892986 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:37.392965668 +0000 UTC m=+235.872853198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : configmap "kube-rbac-proxy" not found Jul 10 00:19:36.893051 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:36.893014 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:37.393000083 +0000 UTC m=+235.872887613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:36.893378 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.893158 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d21b32-6172-4399-889c-ea552b902408-data-volume\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:36.901427 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:36.901399 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvkc5\" (UniqueName: \"kubernetes.io/projected/f0d21b32-6172-4399-889c-ea552b902408-kube-api-access-kvkc5\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:37.396813 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:37.396772 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:37.396993 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:37.396829 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:37.396993 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:37.396888 2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:37.396993 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:37.396922 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:37.396993 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:37.396948 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:38.396933854 +0000 UTC m=+236.876821380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : configmap "kube-rbac-proxy" not found Jul 10 00:19:37.396993 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:37.396978 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:38.396963914 +0000 UTC m=+236.876851441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:38.404980 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:38.404921 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:38.405451 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:38.405088 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:38.405451 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:38.405116 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:38.405451 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:38.405162 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.405142731 +0000 UTC m=+238.885030277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:38.405451 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:38.405194 2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:38.405451 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:38.405229 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:40.405221358 +0000 UTC m=+238.885108884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : configmap "kube-rbac-proxy" not found Jul 10 00:19:38.661914 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:38.661843 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9qxd8_e9015bc2-ec2b-4800-b911-1e190522f37e/dns-node-resolver/0.log" Jul 10 00:19:39.661646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:39.661616 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkxgp_f80f9e53-8206-4724-9677-d1c46b2887f4/node-ca/0.log" Jul 10 00:19:40.262355 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:40.262326 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7ssjm_5c7a8fbe-0641-4a83-91aa-28ce36b25468/serve-healthcheck-canary/0.log" Jul 10 00:19:40.421355 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:40.421327 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:40.421511 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:40.421375 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:40.421511 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:40.421439 2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:40.421511 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:40.421461 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:40.421621 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:40.421523 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:44.421508135 +0000 UTC m=+242.901395694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : configmap "kube-rbac-proxy" not found Jul 10 00:19:40.421621 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:40.421538 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:44.421532357 +0000 UTC m=+242.901419882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:44.453341 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:44.453299 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:44.453822 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:44.453361 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:44.453822 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:44.453425 2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:19:44.453822 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:44.453481 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:52.453467302 +0000 UTC m=+250.933354829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : configmap "kube-rbac-proxy" not found Jul 10 00:19:44.453822 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:44.453479 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:44.453822 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:44.453590 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:52.453569658 +0000 UTC m=+250.933457189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:52.512660 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:52.512622 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:52.513059 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:52.512683 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:52.513059 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:52.512778 2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:19:52.513059 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:19:52.512851 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls podName:f0d21b32-6172-4399-889c-ea552b902408 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:08.512836472 +0000 UTC m=+266.992724001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls") pod "insights-runtime-extractor-b5tqf" (UID: "f0d21b32-6172-4399-889c-ea552b902408") : secret "insights-runtime-extractor-tls" not found Jul 10 00:19:52.513169 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:52.513129 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f0d21b32-6172-4399-889c-ea552b902408-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:19:53.920589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:53.920553 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:19:53.923043 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:53.923017 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f27071-8614-4c45-8f17-ce4587ecd3d2-metrics-certs\") pod \"network-metrics-daemon-vgb4g\" (UID: \"68f27071-8614-4c45-8f17-ce4587ecd3d2\") " pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:19:54.158328 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:54.158299 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qn6qf\"" Jul 10 00:19:54.166411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:54.166393 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vgb4g" Jul 10 00:19:54.286442 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:54.286414 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vgb4g"] Jul 10 00:19:54.289340 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:19:54.289302 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f27071_8614_4c45_8f17_ce4587ecd3d2.slice/crio-8351b23bc1d5c8e69661207ee10b7d08c98f6d972a40868724f404106d6bf50f WatchSource:0}: Error finding container 8351b23bc1d5c8e69661207ee10b7d08c98f6d972a40868724f404106d6bf50f: Status 404 returned error can't find the container with id 8351b23bc1d5c8e69661207ee10b7d08c98f6d972a40868724f404106d6bf50f Jul 10 00:19:54.892684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:54.892644 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgb4g" event={"ID":"68f27071-8614-4c45-8f17-ce4587ecd3d2","Type":"ContainerStarted","Data":"8351b23bc1d5c8e69661207ee10b7d08c98f6d972a40868724f404106d6bf50f"} Jul 10 00:19:55.897429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:55.897392 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgb4g" event={"ID":"68f27071-8614-4c45-8f17-ce4587ecd3d2","Type":"ContainerStarted","Data":"f45727b985a6e54afb7769efa282bc917574d043102b9de2df467336a06a65a9"} Jul 10 00:19:55.897821 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:55.897445 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vgb4g" event={"ID":"68f27071-8614-4c45-8f17-ce4587ecd3d2","Type":"ContainerStarted","Data":"181e8e0a7220fd1b781c0debdacac1b4b0795882997b45083bdcc52725c11904"} Jul 10 00:19:55.912878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:19:55.912824 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vgb4g" podStartSLOduration=252.822299064 podStartE2EDuration="4m13.912806507s" totalImagesPullingTime="1.090507443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:55.911096158 +0000 UTC m=+254.390983706" watchObservedRunningTime="2026-07-10 00:19:55.912806507 +0000 UTC m=+254.392694054" Jul 10 00:20:00.586413 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.586382 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cf9f5cbff-drn4g"] Jul 10 00:20:00.588445 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.588428 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.591091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.591057 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 00:20:00.591388 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.591372 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 00:20:00.592761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.592576 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 00:20:00.593039 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.592867 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s69wz\"" Jul 10 00:20:00.599212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.599193 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 00:20:00.605738 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.605718 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf9f5cbff-drn4g"] Jul 10 00:20:00.668553 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668529 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-image-registry-private-configuration\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668558 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-registry-tls\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668575 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d58d34c-7595-420a-a663-238ff022a889-ca-trust-extracted\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668591 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-registry-certificates\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668622 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bcn\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-kube-api-access-26bcn\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668640 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-trusted-ca\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668686 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-bound-sa-token\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.668956 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.668745 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-installation-pull-secrets\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769548 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-installation-pull-secrets\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769590 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-image-registry-private-configuration\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769608 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-registry-tls\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769626 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d58d34c-7595-420a-a663-238ff022a889-ca-trust-extracted\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769647 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-registry-certificates\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769699 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bcn\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-kube-api-access-26bcn\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769724 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-trusted-ca\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.769750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.769747 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-bound-sa-token\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.770188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.770157 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0d58d34c-7595-420a-a663-238ff022a889-ca-trust-extracted\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.770629 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.770607 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-registry-certificates\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.770750 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.770728 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0d58d34c-7595-420a-a663-238ff022a889-trusted-ca\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.772277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.772256 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-installation-pull-secrets\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.772598 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.772577 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0d58d34c-7595-420a-a663-238ff022a889-image-registry-private-configuration\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.772654 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.772631 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-registry-tls\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.778437 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.778418 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-bound-sa-token\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.778693 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.778675 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bcn\" (UniqueName: \"kubernetes.io/projected/0d58d34c-7595-420a-a663-238ff022a889-kube-api-access-26bcn\") pod \"image-registry-6cf9f5cbff-drn4g\" (UID: \"0d58d34c-7595-420a-a663-238ff022a889\") " pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:00.897883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:00.897793 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:01.038293 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:01.038258 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf9f5cbff-drn4g"] Jul 10 00:20:01.041161 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:01.041127 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58d34c_7595_420a_a663_238ff022a889.slice/crio-e5cdb096bdf800cb1b370db8327a90ee4bc3d0c4afb8affd4ff68e3b36f4d677 WatchSource:0}: Error finding container e5cdb096bdf800cb1b370db8327a90ee4bc3d0c4afb8affd4ff68e3b36f4d677: Status 404 returned error can't find the container with id e5cdb096bdf800cb1b370db8327a90ee4bc3d0c4afb8affd4ff68e3b36f4d677 Jul 10 00:20:01.919430 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:01.919397 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" event={"ID":"0d58d34c-7595-420a-a663-238ff022a889","Type":"ContainerStarted","Data":"10da6df7abaf6f0d4c7d109bcf079fc3149e2c83d04c851ce5a5b38de60128cd"} Jul 10 00:20:01.919878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:01.919448 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" event={"ID":"0d58d34c-7595-420a-a663-238ff022a889","Type":"ContainerStarted","Data":"e5cdb096bdf800cb1b370db8327a90ee4bc3d0c4afb8affd4ff68e3b36f4d677"} Jul 10 00:20:01.919878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:01.919529 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:01.939833 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:01.939785 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podStartSLOduration=1.939770197 podStartE2EDuration="1.939770197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:01.93870342 +0000 UTC m=+260.418590969" watchObservedRunningTime="2026-07-10 00:20:01.939770197 +0000 UTC m=+260.419657745" Jul 10 00:20:07.662361 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.662271 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm"] Jul 10 00:20:07.664800 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.664783 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:07.667175 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.667150 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 00:20:07.667302 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.667283 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rkjsp\"" Jul 10 00:20:07.673088 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.673063 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm"] Jul 10 00:20:07.818325 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.818279 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f8mdm\" (UID: \"6bc04754-ff85-41be-91e8-c2bc59cc4b98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:07.918958 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:07.918872 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f8mdm\" (UID: \"6bc04754-ff85-41be-91e8-c2bc59cc4b98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:07.919081 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:07.919021 2548 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 10 00:20:07.919119 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:07.919096 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates podName:6bc04754-ff85-41be-91e8-c2bc59cc4b98 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:08.419075877 +0000 UTC m=+266.898963407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-f8mdm" (UID: "6bc04754-ff85-41be-91e8-c2bc59cc4b98") : secret "prometheus-operator-admission-webhook-tls" not found Jul 10 00:20:08.422245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.422210 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f8mdm\" (UID: \"6bc04754-ff85-41be-91e8-c2bc59cc4b98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:08.424545 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.424524 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6bc04754-ff85-41be-91e8-c2bc59cc4b98-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f8mdm\" (UID: \"6bc04754-ff85-41be-91e8-c2bc59cc4b98\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:08.522952 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.522918 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:20:08.524912 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.524895 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f0d21b32-6172-4399-889c-ea552b902408-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b5tqf\" (UID: \"f0d21b32-6172-4399-889c-ea552b902408\") " pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:20:08.574137 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.574123 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:08.689301 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.689230 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm"] Jul 10 00:20:08.693347 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:08.693322 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc04754_ff85_41be_91e8_c2bc59cc4b98.slice/crio-da758b59d91f46332553a6a2416d1fc704b404817505fa0d429f4ce933d25e8f WatchSource:0}: Error finding container da758b59d91f46332553a6a2416d1fc704b404817505fa0d429f4ce933d25e8f: Status 404 returned error can't find the container with id da758b59d91f46332553a6a2416d1fc704b404817505fa0d429f4ce933d25e8f Jul 10 00:20:08.790731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.790713 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2znzl\"" Jul 10 00:20:08.798941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.798927 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b5tqf" Jul 10 00:20:08.917776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.917747 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b5tqf"] Jul 10 00:20:08.920854 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:08.920830 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d21b32_6172_4399_889c_ea552b902408.slice/crio-cb1c18777b3e6e429a2aaac973fe29fffc4f4ad350495e80f90b762e20c65c6c WatchSource:0}: Error finding container cb1c18777b3e6e429a2aaac973fe29fffc4f4ad350495e80f90b762e20c65c6c: Status 404 returned error can't find the container with id cb1c18777b3e6e429a2aaac973fe29fffc4f4ad350495e80f90b762e20c65c6c Jul 10 00:20:08.941934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.941515 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" event={"ID":"6bc04754-ff85-41be-91e8-c2bc59cc4b98","Type":"ContainerStarted","Data":"da758b59d91f46332553a6a2416d1fc704b404817505fa0d429f4ce933d25e8f"} Jul 10 00:20:08.945373 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:08.945340 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqf" event={"ID":"f0d21b32-6172-4399-889c-ea552b902408","Type":"ContainerStarted","Data":"cb1c18777b3e6e429a2aaac973fe29fffc4f4ad350495e80f90b762e20c65c6c"} Jul 10 00:20:10.954615 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:10.954559 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" event={"ID":"6bc04754-ff85-41be-91e8-c2bc59cc4b98","Type":"ContainerStarted","Data":"7f58087d6c99898968f9ab0712605edb93f876640ff2d805043b6d9c4dca953b"} Jul 10 00:20:10.955085 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:10.954726 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:10.960164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:10.960138 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" Jul 10 00:20:10.969225 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:10.969040 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f8mdm" podStartSLOduration=2.4808082909999998 podStartE2EDuration="3.969020522s" totalImagesPullingTime="1.488212231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:10.968661435 +0000 UTC m=+269.448548985" watchObservedRunningTime="2026-07-10 00:20:10.969020522 +0000 UTC m=+269.448908071" Jul 10 00:20:12.962947 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:12.962910 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqf" event={"ID":"f0d21b32-6172-4399-889c-ea552b902408","Type":"ContainerStarted","Data":"ffb2899d03510e97fc57a8fbde01938cdeaf20a8936b95f485701951966af16e"} Jul 10 00:20:13.969116 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:13.969084 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqf" event={"ID":"f0d21b32-6172-4399-889c-ea552b902408","Type":"ContainerStarted","Data":"4e4a0d781bbb22f810a4cee2dab1c31cbe23202a7a4f5fb58683a1b307e12d11"} Jul 10 00:20:15.978196 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:15.978149 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b5tqf" event={"ID":"f0d21b32-6172-4399-889c-ea552b902408","Type":"ContainerStarted","Data":"4b4313fe5541deff1580160a075cf9f99a1f639fdef683705ba324aa9e2fb190"} Jul 10 00:20:15.996837 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:15.996788 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b5tqf" podStartSLOduration=33.517086197 podStartE2EDuration="39.996773591s" totalImagesPullingTime="6.479687394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:19:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:15.994735822 +0000 UTC m=+274.474623369" watchObservedRunningTime="2026-07-10 00:20:15.996773591 +0000 UTC m=+274.476661138" Jul 10 00:20:18.077724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.077688 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-n66hc"] Jul 10 00:20:18.079804 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.079788 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.082639 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.082617 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 00:20:18.082765 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.082617 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 00:20:18.082765 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.082622 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 00:20:18.083869 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.083851 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dtj9j\"" Jul 10 00:20:18.083987 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.083925 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 00:20:18.083987 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.083925 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 00:20:18.083987 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.083952 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 00:20:18.092648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.092627 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nrc5s"] Jul 10 00:20:18.094735 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.094719 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.096432 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.096412 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-n66hc"] Jul 10 00:20:18.097078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.097057 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dtnhw\"" Jul 10 00:20:18.097171 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.097084 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 00:20:18.097171 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.097140 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 00:20:18.097269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.097235 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 00:20:18.099311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099293 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.099399 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099331 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.099399 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099358 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.099475 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099434 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b60a2f87-d93a-435a-b087-0bbeccb24351-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.099475 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099455 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvvtx\" (UniqueName: \"kubernetes.io/projected/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-api-access-cvvtx\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.099475 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.099471 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.200779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200731 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.200779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200777 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dddn\" (UniqueName: \"kubernetes.io/projected/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-kube-api-access-5dddn\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200813 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b60a2f87-d93a-435a-b087-0bbeccb24351-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200856 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-sys\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200901 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvvtx\" (UniqueName: \"kubernetes.io/projected/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-api-access-cvvtx\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200924 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-textfile\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200948 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.200994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200971 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-tls\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.200996 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201023 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-wtmp\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201047 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201126 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201160 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-root\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201197 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b60a2f87-d93a-435a-b087-0bbeccb24351-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201198 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-metrics-client-ca\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201247 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.201260 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:18.201260 2548 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 10 00:20:18.201571 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:18.201305 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls podName:b60a2f87-d93a-435a-b087-0bbeccb24351 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:18.701290495 +0000 UTC m=+277.181178025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-n66hc" (UID: "b60a2f87-d93a-435a-b087-0bbeccb24351") : secret "kube-state-metrics-tls" not found Jul 10 00:20:18.201715 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201696 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.201834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.201818 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.203341 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.203322 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.211223 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.211196 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvvtx\" (UniqueName: \"kubernetes.io/projected/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-api-access-cvvtx\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.302619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302584 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302619 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302621 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dddn\" (UniqueName: \"kubernetes.io/projected/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-kube-api-access-5dddn\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302648 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-sys\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302672 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-textfile\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302696 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-tls\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302726 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-sys\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302723 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302764 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-wtmp\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.302826 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302814 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-root\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303147 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302843 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-metrics-client-ca\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303147 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302937 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-wtmp\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303147 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.302989 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-root\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303253 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.303140 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-textfile\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303455 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.303434 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-metrics-client-ca\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.303529 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.303462 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.304941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.304922 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.305030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.305018 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-node-exporter-tls\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.310945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.310921 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dddn\" (UniqueName: \"kubernetes.io/projected/bb70b152-4671-4cf1-bcb7-956cc02bd0f8-kube-api-access-5dddn\") pod \"node-exporter-nrc5s\" (UID: \"bb70b152-4671-4cf1-bcb7-956cc02bd0f8\") " pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.357806 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.357726 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:18.360675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.360660 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.363195 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363163 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 00:20:18.363467 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363425 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7w2x\"" Jul 10 00:20:18.363467 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363443 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 00:20:18.363641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363532 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 00:20:18.363700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363643 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 00:20:18.363700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363643 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 00:20:18.363891 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363877 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 00:20:18.363961 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.363943 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 00:20:18.369281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.369259 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 00:20:18.369915 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.369894 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:18.403551 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403515 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nrc5s" Jul 10 00:20:18.403726 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403627 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403726 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403693 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403787 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403814 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403841 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403980 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403880 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzkw\" (UniqueName: \"kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.403980 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.403925 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.412443 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:18.412414 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb70b152_4671_4cf1_bcb7_956cc02bd0f8.slice/crio-1441af63ef0fdab4f1a61245761283afba2faf6a5ea469505b99802293b52b77 WatchSource:0}: Error finding container 1441af63ef0fdab4f1a61245761283afba2faf6a5ea469505b99802293b52b77: Status 404 returned error can't find the container with id 1441af63ef0fdab4f1a61245761283afba2faf6a5ea469505b99802293b52b77 Jul 10 00:20:18.505016 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.504962 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505039 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505100 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505317 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505281 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505445 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505400 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505445 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505428 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzkw\" (UniqueName: \"kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505573 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505478 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.505900 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505879 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.506007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.505977 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.506094 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.506077 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.506177 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.506158 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.507713 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.507684 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.507831 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.507814 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.513977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.513955 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzkw\" (UniqueName: \"kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw\") pod \"console-57f95674d8-mqsd7\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.671541 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.671426 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:18.708397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.708351 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.710687 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.710661 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b60a2f87-d93a-435a-b087-0bbeccb24351-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-n66hc\" (UID: \"b60a2f87-d93a-435a-b087-0bbeccb24351\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.795783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.795751 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:18.798673 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:18.798646 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c687fda_c16e_4756_aacf_f4e7d055c289.slice/crio-c2503ec84aa406c30c059d23a46bd37737c5884b52dd66c92892c05b32960862 WatchSource:0}: Error finding container c2503ec84aa406c30c059d23a46bd37737c5884b52dd66c92892c05b32960862: Status 404 returned error can't find the container with id c2503ec84aa406c30c059d23a46bd37737c5884b52dd66c92892c05b32960862 Jul 10 00:20:18.988603 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.988566 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" Jul 10 00:20:18.990064 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.990036 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrc5s" event={"ID":"bb70b152-4671-4cf1-bcb7-956cc02bd0f8","Type":"ContainerStarted","Data":"1441af63ef0fdab4f1a61245761283afba2faf6a5ea469505b99802293b52b77"} Jul 10 00:20:18.991641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:18.991614 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f95674d8-mqsd7" event={"ID":"4c687fda-c16e-4756-aacf-f4e7d055c289","Type":"ContainerStarted","Data":"c2503ec84aa406c30c059d23a46bd37737c5884b52dd66c92892c05b32960862"} Jul 10 00:20:19.122183 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.122153 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:20:19.125473 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.125451 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-n66hc"] Jul 10 00:20:19.125625 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.125587 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.128473 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128448 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:20:19.128590 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128474 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:20:19.128590 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128509 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:20:19.128590 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128547 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:20:19.128738 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128600 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:20:19.128839 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128817 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:20:19.128950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128886 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tp8xp\"" Jul 10 00:20:19.128950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128907 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:20:19.128950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128929 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:20:19.129143 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.128988 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:20:19.129318 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:19.129293 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60a2f87_d93a_435a_b087_0bbeccb24351.slice/crio-b016e6141226f73adcb76b631ff4f45c0631d20f988cd32cfe3a418363dd9d13 WatchSource:0}: Error finding container b016e6141226f73adcb76b631ff4f45c0631d20f988cd32cfe3a418363dd9d13: Status 404 returned error can't find the container with id b016e6141226f73adcb76b631ff4f45c0631d20f988cd32cfe3a418363dd9d13 Jul 10 00:20:19.145433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.145406 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:20:19.212881 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212846 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212890 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212927 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212956 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212978 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.212998 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213049 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213023 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213111 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213147 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5n4\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213222 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213250 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213277 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.213331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.213299 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314339 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314373 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314392 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314413 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314441 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314469 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314471 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314526 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314550 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314572 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.314894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.314598 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.315187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.315157 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.315187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.315175 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.315364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.315219 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.315364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.315249 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5n4\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.317123 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.317094 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.317897 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.317644 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.323106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.320187 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.323106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.321562 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.323106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.322860 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.323403 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.323376 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.323704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.323627 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.324521 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.323872 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.324521 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.323954 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.326863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.324819 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.326863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.325591 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5n4\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.326863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.326190 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.436261 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.436209 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:20:19.645976 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.645957 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:20:19.648047 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:19.648020 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fc6c99_70c0_499d_9356_cc68a645e9ca.slice/crio-1de3310a854ac7096b53a3b9ed133b83ec90fe068a64e8a2660266534bb139df WatchSource:0}: Error finding container 1de3310a854ac7096b53a3b9ed133b83ec90fe068a64e8a2660266534bb139df: Status 404 returned error can't find the container with id 1de3310a854ac7096b53a3b9ed133b83ec90fe068a64e8a2660266534bb139df Jul 10 00:20:19.997614 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.997569 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"1de3310a854ac7096b53a3b9ed133b83ec90fe068a64e8a2660266534bb139df"} Jul 10 00:20:19.999218 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:19.999185 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" event={"ID":"b60a2f87-d93a-435a-b087-0bbeccb24351","Type":"ContainerStarted","Data":"b016e6141226f73adcb76b631ff4f45c0631d20f988cd32cfe3a418363dd9d13"} Jul 10 00:20:20.001217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:20.001188 2548 generic.go:356] "Generic (PLEG): container finished" podID="bb70b152-4671-4cf1-bcb7-956cc02bd0f8" containerID="e00bb586136bb9e9eb769f07849ae795ea5f579ffdc4b1a36cd37fb428f36486" exitCode=0 Jul 10 00:20:20.001352 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:20.001245 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrc5s" event={"ID":"bb70b152-4671-4cf1-bcb7-956cc02bd0f8","Type":"ContainerDied","Data":"e00bb586136bb9e9eb769f07849ae795ea5f579ffdc4b1a36cd37fb428f36486"} Jul 10 00:20:20.904648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:20.904598 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:20.905089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:20.904670 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:21.012684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.012643 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" event={"ID":"b60a2f87-d93a-435a-b087-0bbeccb24351","Type":"ContainerStarted","Data":"0d3cd7395e0db43a9dc979f1127e1efef6751cf157c6bf9faec360b8b0a1d73b"} Jul 10 00:20:21.012866 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.012718 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" event={"ID":"b60a2f87-d93a-435a-b087-0bbeccb24351","Type":"ContainerStarted","Data":"e198ee57c6aec794f024ca51bab41307c8521a929e9bc61e41b8049db847a16d"} Jul 10 00:20:21.012866 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.012744 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" event={"ID":"b60a2f87-d93a-435a-b087-0bbeccb24351","Type":"ContainerStarted","Data":"e3d366a1cc3de87518cb727d413027959dd271dd54bfa0422f2bf9f85f404142"} Jul 10 00:20:21.015681 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.015652 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrc5s" event={"ID":"bb70b152-4671-4cf1-bcb7-956cc02bd0f8","Type":"ContainerStarted","Data":"a9b4229705999f51b6e2fc7c7c661e4758760f595f3187e3ea8a5991d87d2ed3"} Jul 10 00:20:21.015823 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.015709 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nrc5s" event={"ID":"bb70b152-4671-4cf1-bcb7-956cc02bd0f8","Type":"ContainerStarted","Data":"da700fedba9d801f5fb2598443164363507aced6e8b20400cd444d1ea4f28f59"} Jul 10 00:20:21.034552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.034477 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-n66hc" podStartSLOduration=1.734880303 podStartE2EDuration="3.034455308s" totalImagesPullingTime="1.299575005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:21.031706296 +0000 UTC m=+279.511593847" watchObservedRunningTime="2026-07-10 00:20:21.034455308 +0000 UTC m=+279.514342858" Jul 10 00:20:21.057272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:21.057215 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nrc5s" podStartSLOduration=1.902367921 podStartE2EDuration="3.057194736s" totalImagesPullingTime="1.08121879s" totalInitContainerRuntime="73.608025ms" isStatefulPod=true podCreationTimestamp="2026-07-10 00:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:21.056744071 +0000 UTC m=+279.536631632" watchObservedRunningTime="2026-07-10 00:20:21.057194736 +0000 UTC m=+279.537082278" Jul 10 00:20:22.928201 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:22.928164 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:22.928602 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:22.928222 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:23.023604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.023573 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995" exitCode=0 Jul 10 00:20:23.023778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.023662 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995"} Jul 10 00:20:23.025297 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.025276 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f95674d8-mqsd7" event={"ID":"4c687fda-c16e-4756-aacf-f4e7d055c289","Type":"ContainerStarted","Data":"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5"} Jul 10 00:20:23.069846 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.069794 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57f95674d8-mqsd7" podStartSLOduration=1.58828277 podStartE2EDuration="5.069778402s" totalImagesPullingTime="3.481495632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:23.068820016 +0000 UTC m=+281.548707565" watchObservedRunningTime="2026-07-10 00:20:23.069778402 +0000 UTC m=+281.549665950" Jul 10 00:20:23.260105 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.260070 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm"] Jul 10 00:20:23.263465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.263449 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.266127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266100 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 00:20:23.266232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266183 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 00:20:23.266232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266201 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 00:20:23.266232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266218 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 00:20:23.266232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266187 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nxld8\"" Jul 10 00:20:23.266584 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.266568 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 00:20:23.271055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.271034 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 00:20:23.275332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.275313 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm"] Jul 10 00:20:23.353623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353600 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-metrics-client-ca\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353710 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353626 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353710 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353656 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-federate-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353718 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-serving-certs-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353743 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353767 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353801 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdgf\" (UniqueName: \"kubernetes.io/projected/427040be-9b5c-4ae6-a1c2-254378bd7371-kube-api-access-jqdgf\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.353901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.353833 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454241 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdgf\" (UniqueName: \"kubernetes.io/projected/427040be-9b5c-4ae6-a1c2-254378bd7371-kube-api-access-jqdgf\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454300 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454343 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-metrics-client-ca\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454367 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454421 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-federate-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454462 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-serving-certs-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454487 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.454606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.454540 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.455181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.455156 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-metrics-client-ca\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.455281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.455196 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-serving-certs-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.455343 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.455300 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.457006 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.456986 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.457085 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.457058 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-federate-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.457302 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.457286 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-telemeter-client-tls\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.457335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.457318 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/427040be-9b5c-4ae6-a1c2-254378bd7371-secret-telemeter-client\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.461808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.461786 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdgf\" (UniqueName: \"kubernetes.io/projected/427040be-9b5c-4ae6-a1c2-254378bd7371-kube-api-access-jqdgf\") pod \"telemeter-client-5f87c7cbff-hsgqm\" (UID: \"427040be-9b5c-4ae6-a1c2-254378bd7371\") " pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.573537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.573447 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" Jul 10 00:20:23.732075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:23.731977 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm"] Jul 10 00:20:23.735181 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:23.735156 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427040be_9b5c_4ae6_a1c2_254378bd7371.slice/crio-918acf8fd40d8eebe3f8e41ed51ba39eaaac85f0f8f2c4e8d3ef0b962332ddb9 WatchSource:0}: Error finding container 918acf8fd40d8eebe3f8e41ed51ba39eaaac85f0f8f2c4e8d3ef0b962332ddb9: Status 404 returned error can't find the container with id 918acf8fd40d8eebe3f8e41ed51ba39eaaac85f0f8f2c4e8d3ef0b962332ddb9 Jul 10 00:20:24.029451 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:24.029418 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" event={"ID":"427040be-9b5c-4ae6-a1c2-254378bd7371","Type":"ContainerStarted","Data":"918acf8fd40d8eebe3f8e41ed51ba39eaaac85f0f8f2c4e8d3ef0b962332ddb9"} Jul 10 00:20:24.636198 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:24.636150 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z5hkd" podUID="0559a2bd-a537-451d-ba80-a6a74fbf2067" Jul 10 00:20:25.035680 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:25.035647 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8"} Jul 10 00:20:25.035680 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:25.035670 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:26.041605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.041527 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" event={"ID":"427040be-9b5c-4ae6-a1c2-254378bd7371","Type":"ContainerStarted","Data":"43c7f262f08531914be931d082c4186093cc42abdfabb88b640eaacfe2c6c8ef"} Jul 10 00:20:26.041605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.041585 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" event={"ID":"427040be-9b5c-4ae6-a1c2-254378bd7371","Type":"ContainerStarted","Data":"7746741ae3966dc27290b89843634ba44466eb2177c7c44089f10da76ae06a37"} Jul 10 00:20:26.044453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.044425 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a"} Jul 10 00:20:26.044574 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.044507 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea"} Jul 10 00:20:26.044574 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.044537 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5"} Jul 10 00:20:26.044574 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:26.044562 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40"} Jul 10 00:20:27.051025 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.050994 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerStarted","Data":"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4"} Jul 10 00:20:27.052884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.052858 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" event={"ID":"427040be-9b5c-4ae6-a1c2-254378bd7371","Type":"ContainerStarted","Data":"df46486d22d49d70d66cdf8f3873b4173a6590ecd7154cb29f3238e517292678"} Jul 10 00:20:27.076828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.076775 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6941490200000002 podStartE2EDuration="8.076761715s" totalImagesPullingTime="5.36828445s" totalInitContainerRuntime="14.328245ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:27.075133555 +0000 UTC m=+285.555021104" watchObservedRunningTime="2026-07-10 00:20:27.076761715 +0000 UTC m=+285.556649262" Jul 10 00:20:27.096007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.095958 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f87c7cbff-hsgqm" podStartSLOduration=2.067706238 podStartE2EDuration="4.095936637s" totalImagesPullingTime="2.028230399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:27.094514924 +0000 UTC m=+285.574402474" watchObservedRunningTime="2026-07-10 00:20:27.095936637 +0000 UTC m=+285.575824198" Jul 10 00:20:27.179271 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.179241 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:27.893899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.893859 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:27.896149 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:27.896129 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0559a2bd-a537-451d-ba80-a6a74fbf2067-metrics-tls\") pod \"dns-default-z5hkd\" (UID: \"0559a2bd-a537-451d-ba80-a6a74fbf2067\") " pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:28.039253 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:28.039223 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-crl8x\"" Jul 10 00:20:28.047579 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:28.047561 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:28.170603 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:28.170534 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5hkd"] Jul 10 00:20:28.174614 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:20:28.174590 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0559a2bd_a537_451d_ba80_a6a74fbf2067.slice/crio-915c1a1788b31d2880687ba2615c57b10d8e07325909d7e3c68454986b569821 WatchSource:0}: Error finding container 915c1a1788b31d2880687ba2615c57b10d8e07325909d7e3c68454986b569821: Status 404 returned error can't find the container with id 915c1a1788b31d2880687ba2615c57b10d8e07325909d7e3c68454986b569821 Jul 10 00:20:28.672150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:28.672114 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:29.062477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:29.062440 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5hkd" event={"ID":"0559a2bd-a537-451d-ba80-a6a74fbf2067","Type":"ContainerStarted","Data":"915c1a1788b31d2880687ba2615c57b10d8e07325909d7e3c68454986b569821"} Jul 10 00:20:30.069876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:30.069844 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5hkd" event={"ID":"0559a2bd-a537-451d-ba80-a6a74fbf2067","Type":"ContainerStarted","Data":"4b171c8c7ef2446eea1edd20b221994fbd43d5544a3222b2eebe2e566916a0d4"} Jul 10 00:20:30.901741 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:30.901704 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:30.901902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:30.901754 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:31.075472 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:31.075436 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5hkd" event={"ID":"0559a2bd-a537-451d-ba80-a6a74fbf2067","Type":"ContainerStarted","Data":"5f425a73cffd1eab663504bbb3cfab2bbadb3a6864a758398171f205c4ad7cee"} Jul 10 00:20:31.075841 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:31.075651 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:31.091906 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:31.091853 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z5hkd" podStartSLOduration=252.333309484 podStartE2EDuration="4m14.091837964s" totalImagesPullingTime="1.75852848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:16:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:20:31.090857352 +0000 UTC m=+289.570744900" watchObservedRunningTime="2026-07-10 00:20:31.091837964 +0000 UTC m=+289.571725547" Jul 10 00:20:32.928127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:32.928091 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:32.928534 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:32.928142 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:40.901983 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.901944 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:40.902462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.902009 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:40.902462 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.902047 2548 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:40.902566 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.902468 2548 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"10da6df7abaf6f0d4c7d109bcf079fc3149e2c83d04c851ce5a5b38de60128cd"} pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" containerMessage="Container registry failed liveness probe, will be restarted" Jul 10 00:20:40.906090 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.906059 2548 patch_prober.go:28] interesting pod/image-registry-6cf9f5cbff-drn4g container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:20:40.906215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:40.906149 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:20:41.081456 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:41.081427 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5hkd" Jul 10 00:20:42.000778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:42.000750 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:20:42.000778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:42.000750 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:20:42.003771 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:42.003753 2548 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 00:20:50.906736 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:50.906708 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:20:52.199569 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.199510 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57f95674d8-mqsd7" podUID="4c687fda-c16e-4756-aacf-f4e7d055c289" containerName="console" containerID="cri-o://026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5" gracePeriod=15 Jul 10 00:20:52.438538 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.438505 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:52.595014 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.594973 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595014 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595014 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595035 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595077 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595131 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595154 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwzkw\" (UniqueName: \"kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595192 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert\") pod \"4c687fda-c16e-4756-aacf-f4e7d055c289\" (UID: \"4c687fda-c16e-4756-aacf-f4e7d055c289\") " Jul 10 00:20:52.595540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595455 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config" (OuterVolumeSpecName: "console-config") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:52.595604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595546 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:52.595678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595654 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca" (OuterVolumeSpecName: "service-ca") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:52.595725 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.595670 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:20:52.597605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.597509 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:20:52.597605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.597575 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:20:52.597748 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.597602 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw" (OuterVolumeSpecName: "kube-api-access-zwzkw") pod "4c687fda-c16e-4756-aacf-f4e7d055c289" (UID: "4c687fda-c16e-4756-aacf-f4e7d055c289"). InnerVolumeSpecName "kube-api-access-zwzkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:20:52.696480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696427 2548 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-oauth-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696467 2548 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-trusted-ca-bundle\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696477 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwzkw\" (UniqueName: \"kubernetes.io/projected/4c687fda-c16e-4756-aacf-f4e7d055c289-kube-api-access-zwzkw\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696486 2548 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c687fda-c16e-4756-aacf-f4e7d055c289-console-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696519 2548 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-service-ca\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696528 2548 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-console-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:52.696796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:52.696535 2548 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c687fda-c16e-4756-aacf-f4e7d055c289-oauth-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:20:53.157409 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.157376 2548 generic.go:356] "Generic (PLEG): container finished" podID="4c687fda-c16e-4756-aacf-f4e7d055c289" containerID="026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5" exitCode=0 Jul 10 00:20:53.157609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.157422 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57f95674d8-mqsd7" Jul 10 00:20:53.157609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.157461 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f95674d8-mqsd7" event={"ID":"4c687fda-c16e-4756-aacf-f4e7d055c289","Type":"ContainerDied","Data":"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5"} Jul 10 00:20:53.157609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.157523 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57f95674d8-mqsd7" event={"ID":"4c687fda-c16e-4756-aacf-f4e7d055c289","Type":"ContainerDied","Data":"c2503ec84aa406c30c059d23a46bd37737c5884b52dd66c92892c05b32960862"} Jul 10 00:20:53.157609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.157544 2548 scope.go:117] "RemoveContainer" containerID="026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5" Jul 10 00:20:53.166834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.166812 2548 scope.go:117] "RemoveContainer" containerID="026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5" Jul 10 00:20:53.167145 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:20:53.167125 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5\": container with ID starting with 026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5 not found: ID does not exist" containerID="026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5" Jul 10 00:20:53.167224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.167153 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5"} err="failed to get container status \"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5\": rpc error: code = NotFound desc = could not find container \"026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5\": container with ID starting with 026a9613b372ababf17ce08ef2949e70d1eade5e52b969731861d60b5e40b4e5 not found: ID does not exist" Jul 10 00:20:53.178686 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.178654 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:53.182531 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:53.182504 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57f95674d8-mqsd7"] Jul 10 00:20:54.160953 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:20:54.158116 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c687fda-c16e-4756-aacf-f4e7d055c289" path="/var/lib/kubelet/pods/4c687fda-c16e-4756-aacf-f4e7d055c289/volumes" Jul 10 00:21:00.913644 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:00.913605 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" podUID="d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:21:05.922642 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:05.922601 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" podUID="0d58d34c-7595-420a-a663-238ff022a889" containerName="registry" containerID="cri-o://10da6df7abaf6f0d4c7d109bcf079fc3149e2c83d04c851ce5a5b38de60128cd" gracePeriod=30 Jul 10 00:21:07.212715 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:07.212676 2548 generic.go:356] "Generic (PLEG): container finished" podID="0d58d34c-7595-420a-a663-238ff022a889" containerID="10da6df7abaf6f0d4c7d109bcf079fc3149e2c83d04c851ce5a5b38de60128cd" exitCode=0 Jul 10 00:21:07.213096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:07.212722 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" event={"ID":"0d58d34c-7595-420a-a663-238ff022a889","Type":"ContainerDied","Data":"10da6df7abaf6f0d4c7d109bcf079fc3149e2c83d04c851ce5a5b38de60128cd"} Jul 10 00:21:07.213096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:07.212832 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:21:07.213096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:07.212855 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" event={"ID":"0d58d34c-7595-420a-a663-238ff022a889","Type":"ContainerStarted","Data":"a3fe141b1927381bf9599e159c1339ca2088c05c0346d2b97d079d4a5f75d10e"} Jul 10 00:21:10.913727 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:10.913686 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" podUID="d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:21:12.666287 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:12.666259 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cf9f5cbff-drn4g_0d58d34c-7595-420a-a663-238ff022a889/registry/0.log" Jul 10 00:21:12.725517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:12.725481 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cf9f5cbff-drn4g_0d58d34c-7595-420a-a663-238ff022a889/registry/1.log" Jul 10 00:21:13.126269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:13.126239 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkxgp_f80f9e53-8206-4724-9677-d1c46b2887f4/node-ca/0.log" Jul 10 00:21:13.728166 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:13.728135 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7ssjm_5c7a8fbe-0641-4a83-91aa-28ce36b25468/serve-healthcheck-canary/0.log" Jul 10 00:21:20.913475 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:20.913426 2548 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" podUID="d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:21:20.913863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:20.913527 2548 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" Jul 10 00:21:20.914091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:20.914071 2548 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"84b8a8e615f8191d0eb35f3588d9402ca863f6b11a34760b938542e4b761ad30"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 00:21:20.914127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:20.914113 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" podUID="d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92" containerName="service-proxy" containerID="cri-o://84b8a8e615f8191d0eb35f3588d9402ca863f6b11a34760b938542e4b761ad30" gracePeriod=30 Jul 10 00:21:21.267935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:21.267897 2548 generic.go:356] "Generic (PLEG): container finished" podID="d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92" containerID="84b8a8e615f8191d0eb35f3588d9402ca863f6b11a34760b938542e4b761ad30" exitCode=2 Jul 10 00:21:21.268087 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:21.267964 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerDied","Data":"84b8a8e615f8191d0eb35f3588d9402ca863f6b11a34760b938542e4b761ad30"} Jul 10 00:21:21.268087 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:21.268009 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f965c669c-ghgqv" event={"ID":"d4fd94ff-8d8c-4f49-bd5d-84d9daca9b92","Type":"ContainerStarted","Data":"241bc01742b3d37733776a0513fa03bba309c23a52ec1810e822fd9d1cf1404e"} Jul 10 00:21:28.220407 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:28.220375 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cf9f5cbff-drn4g" Jul 10 00:21:38.322033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.321952 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:38.322405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322242 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="alertmanager" containerID="cri-o://d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" gracePeriod=120 Jul 10 00:21:38.322405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322302 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-metric" containerID="cri-o://3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" gracePeriod=120 Jul 10 00:21:38.322405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322356 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="config-reloader" containerID="cri-o://ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" gracePeriod=120 Jul 10 00:21:38.322628 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322319 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-web" containerID="cri-o://856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" gracePeriod=120 Jul 10 00:21:38.322628 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322320 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy" containerID="cri-o://6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" gracePeriod=120 Jul 10 00:21:38.322628 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:38.322393 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="prom-label-proxy" containerID="cri-o://a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" gracePeriod=120 Jul 10 00:21:39.341349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341315 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" exitCode=0 Jul 10 00:21:39.341349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341340 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" exitCode=0 Jul 10 00:21:39.341349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341347 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" exitCode=0 Jul 10 00:21:39.341349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341353 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" exitCode=0 Jul 10 00:21:39.341819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341380 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4"} Jul 10 00:21:39.341819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341439 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea"} Jul 10 00:21:39.341819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341465 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40"} Jul 10 00:21:39.341819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.341505 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8"} Jul 10 00:21:39.555244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.555217 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:39.653111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653030 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653066 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653108 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653183 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653228 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653251 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653273 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653296 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653317 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t5n4\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653343 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653373 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653423 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653448 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config\") pod \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\" (UID: \"d1fc6c99-70c0-499d-9356-cc68a645e9ca\") " Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653458 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653462 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653700 2548 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.653812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653720 2548 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-alertmanager-main-db\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.654116 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.653876 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:21:39.656113 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.656088 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out" (OuterVolumeSpecName: "config-out") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:21:39.656606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.656576 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:21:39.656790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.656746 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.656902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.656843 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.657054 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.657022 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.657157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.657066 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.657558 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.657534 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4" (OuterVolumeSpecName: "kube-api-access-4t5n4") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "kube-api-access-4t5n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:21:39.658252 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.658235 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume" (OuterVolumeSpecName: "config-volume") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.660480 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.660415 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.666278 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.666254 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config" (OuterVolumeSpecName: "web-config") pod "d1fc6c99-70c0-499d-9356-cc68a645e9ca" (UID: "d1fc6c99-70c0-499d-9356-cc68a645e9ca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:21:39.754244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754207 2548 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-cluster-tls-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754236 2548 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-volume\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754247 2548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754258 2548 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1fc6c99-70c0-499d-9356-cc68a645e9ca-metrics-client-ca\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754268 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5n4\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-kube-api-access-4t5n4\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754278 2548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754288 2548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-main-tls\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754298 2548 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754306 2548 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1fc6c99-70c0-499d-9356-cc68a645e9ca-web-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754314 2548 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1fc6c99-70c0-499d-9356-cc68a645e9ca-config-out\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:39.754440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:39.754322 2548 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1fc6c99-70c0-499d-9356-cc68a645e9ca-tls-assets\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:21:40.347567 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347536 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" exitCode=0 Jul 10 00:21:40.347567 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347563 2548 generic.go:356] "Generic (PLEG): container finished" podID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerID="856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" exitCode=0 Jul 10 00:21:40.347981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347606 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.347981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347618 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a"} Jul 10 00:21:40.347981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347689 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5"} Jul 10 00:21:40.347981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347718 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1fc6c99-70c0-499d-9356-cc68a645e9ca","Type":"ContainerDied","Data":"1de3310a854ac7096b53a3b9ed133b83ec90fe068a64e8a2660266534bb139df"} Jul 10 00:21:40.347981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.347757 2548 scope.go:117] "RemoveContainer" containerID="a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" Jul 10 00:21:40.357964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.355925 2548 scope.go:117] "RemoveContainer" containerID="3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" Jul 10 00:21:40.364524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.364484 2548 scope.go:117] "RemoveContainer" containerID="6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" Jul 10 00:21:40.370999 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.370933 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:40.371378 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.371355 2548 scope.go:117] "RemoveContainer" containerID="856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" Jul 10 00:21:40.375405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.375384 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:40.379232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.379218 2548 scope.go:117] "RemoveContainer" containerID="ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" Jul 10 00:21:40.387565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.387550 2548 scope.go:117] "RemoveContainer" containerID="d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" Jul 10 00:21:40.394235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.394213 2548 scope.go:117] "RemoveContainer" containerID="a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995" Jul 10 00:21:40.397929 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.397907 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:40.398212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398193 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="config-reloader" Jul 10 00:21:40.398212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398211 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="config-reloader" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398222 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="init-config-reloader" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398229 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="init-config-reloader" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398235 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398241 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398246 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="prom-label-proxy" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398251 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="prom-label-proxy" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398259 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-metric" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398264 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-metric" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398274 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-web" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398282 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-web" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398292 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c687fda-c16e-4756-aacf-f4e7d055c289" containerName="console" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398297 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c687fda-c16e-4756-aacf-f4e7d055c289" containerName="console" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398305 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="alertmanager" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398310 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="alertmanager" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398351 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="alertmanager" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398361 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398369 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-metric" Jul 10 00:21:40.398368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398377 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="prom-label-proxy" Jul 10 00:21:40.399001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398382 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c687fda-c16e-4756-aacf-f4e7d055c289" containerName="console" Jul 10 00:21:40.399001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398387 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="config-reloader" Jul 10 00:21:40.399001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.398392 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" containerName="kube-rbac-proxy-web" Jul 10 00:21:40.401434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.401414 2548 scope.go:117] "RemoveContainer" containerID="a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" Jul 10 00:21:40.401767 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.401749 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4\": container with ID starting with a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4 not found: ID does not exist" containerID="a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" Jul 10 00:21:40.401867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.401775 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4"} err="failed to get container status \"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4\": rpc error: code = NotFound desc = could not find container \"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4\": container with ID starting with a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4 not found: ID does not exist" Jul 10 00:21:40.401867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.401793 2548 scope.go:117] "RemoveContainer" containerID="3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" Jul 10 00:21:40.402049 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.402028 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a\": container with ID starting with 3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a not found: ID does not exist" containerID="3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" Jul 10 00:21:40.402086 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402056 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a"} err="failed to get container status \"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a\": rpc error: code = NotFound desc = could not find container \"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a\": container with ID starting with 3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a not found: ID does not exist" Jul 10 00:21:40.402086 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402072 2548 scope.go:117] "RemoveContainer" containerID="6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" Jul 10 00:21:40.402281 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.402267 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea\": container with ID starting with 6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea not found: ID does not exist" containerID="6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" Jul 10 00:21:40.402321 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402284 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea"} err="failed to get container status \"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea\": rpc error: code = NotFound desc = could not find container \"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea\": container with ID starting with 6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea not found: ID does not exist" Jul 10 00:21:40.402321 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402295 2548 scope.go:117] "RemoveContainer" containerID="856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" Jul 10 00:21:40.402482 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.402469 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5\": container with ID starting with 856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5 not found: ID does not exist" containerID="856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" Jul 10 00:21:40.402538 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402485 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5"} err="failed to get container status \"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5\": rpc error: code = NotFound desc = could not find container \"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5\": container with ID starting with 856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5 not found: ID does not exist" Jul 10 00:21:40.402538 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402513 2548 scope.go:117] "RemoveContainer" containerID="ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" Jul 10 00:21:40.402697 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.402683 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40\": container with ID starting with ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40 not found: ID does not exist" containerID="ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" Jul 10 00:21:40.402737 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402699 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40"} err="failed to get container status \"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40\": rpc error: code = NotFound desc = could not find container \"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40\": container with ID starting with ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40 not found: ID does not exist" Jul 10 00:21:40.402737 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402711 2548 scope.go:117] "RemoveContainer" containerID="d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" Jul 10 00:21:40.402910 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.402896 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8\": container with ID starting with d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8 not found: ID does not exist" containerID="d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" Jul 10 00:21:40.402958 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402911 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8"} err="failed to get container status \"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8\": rpc error: code = NotFound desc = could not find container \"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8\": container with ID starting with d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8 not found: ID does not exist" Jul 10 00:21:40.402958 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.402922 2548 scope.go:117] "RemoveContainer" containerID="a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995" Jul 10 00:21:40.403136 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:21:40.403122 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995\": container with ID starting with a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995 not found: ID does not exist" containerID="a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995" Jul 10 00:21:40.403177 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403136 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995"} err="failed to get container status \"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995\": rpc error: code = NotFound desc = could not find container \"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995\": container with ID starting with a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995 not found: ID does not exist" Jul 10 00:21:40.403177 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403147 2548 scope.go:117] "RemoveContainer" containerID="a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4" Jul 10 00:21:40.403355 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403337 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4"} err="failed to get container status \"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4\": rpc error: code = NotFound desc = could not find container \"a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4\": container with ID starting with a51d9d0ed6e9d46cbbc94c96e84e881b8fe998749b57645450129b24565179d4 not found: ID does not exist" Jul 10 00:21:40.403396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403355 2548 scope.go:117] "RemoveContainer" containerID="3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a" Jul 10 00:21:40.403440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403426 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.403724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403694 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a"} err="failed to get container status \"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a\": rpc error: code = NotFound desc = could not find container \"3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a\": container with ID starting with 3d231c316b8c5f24935f3e36a411862a30948741bca58d38e3cb91d0efd3af6a not found: ID does not exist" Jul 10 00:21:40.403724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403724 2548 scope.go:117] "RemoveContainer" containerID="6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea" Jul 10 00:21:40.403964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403943 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea"} err="failed to get container status \"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea\": rpc error: code = NotFound desc = could not find container \"6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea\": container with ID starting with 6636f0a7163017e67e7dac0a9d762db4d22ab94228880e087a63f4be37dfd4ea not found: ID does not exist" Jul 10 00:21:40.404033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.403966 2548 scope.go:117] "RemoveContainer" containerID="856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5" Jul 10 00:21:40.404221 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404198 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5"} err="failed to get container status \"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5\": rpc error: code = NotFound desc = could not find container \"856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5\": container with ID starting with 856d8b73bd3d90129b4a1c1f8bad9389a8e639ce22fd50c736631bed6f1151a5 not found: ID does not exist" Jul 10 00:21:40.404271 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404225 2548 scope.go:117] "RemoveContainer" containerID="ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40" Jul 10 00:21:40.404465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404448 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40"} err="failed to get container status \"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40\": rpc error: code = NotFound desc = could not find container \"ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40\": container with ID starting with ff5de782fa20cdab4da91015cf0599608703e2480f7faa54b74e25295c80de40 not found: ID does not exist" Jul 10 00:21:40.404465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404464 2548 scope.go:117] "RemoveContainer" containerID="d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8" Jul 10 00:21:40.404712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404694 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8"} err="failed to get container status \"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8\": rpc error: code = NotFound desc = could not find container \"d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8\": container with ID starting with d583f9889cace96e6f6d1987405fdeb6239f03aceb2d79148e50ce6b9a7f96a8 not found: ID does not exist" Jul 10 00:21:40.404712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404711 2548 scope.go:117] "RemoveContainer" containerID="a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995" Jul 10 00:21:40.404928 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.404913 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995"} err="failed to get container status \"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995\": rpc error: code = NotFound desc = could not find container \"a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995\": container with ID starting with a4d6d80ae2af624c007d3ec4f5fadcc9ce2e13476ad807f851ac0727b3957995 not found: ID does not exist" Jul 10 00:21:40.406270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406250 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:21:40.406362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406268 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:21:40.406362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406250 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:21:40.406362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406355 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:21:40.406530 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406481 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:21:40.406530 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406524 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:21:40.406627 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406612 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tp8xp\"" Jul 10 00:21:40.406727 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406711 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:21:40.407007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.406991 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:21:40.411407 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.411387 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:21:40.416768 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.416718 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:40.458908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.458877 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.458908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.458908 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.458937 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcpmc\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-kube-api-access-qcpmc\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459017 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459069 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459130 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459154 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459171 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459227 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459248 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459274 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459301 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.459404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.459322 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.521901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.521858 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:21:40.526115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.526091 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.529119 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529097 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 00:21:40.529220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529155 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 00:21:40.529220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529158 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 00:21:40.529344 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529217 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 00:21:40.529344 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529290 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g7w2x\"" Jul 10 00:21:40.529344 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529291 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 00:21:40.529344 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.529319 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 00:21:40.530670 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.530656 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 00:21:40.535810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.535787 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 00:21:40.537137 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.537115 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:21:40.560192 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560157 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560205 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560228 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560282 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcpmc\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-kube-api-access-qcpmc\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560326 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560363 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560387 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560411 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560438 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560522 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560553 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560586 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.560851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.560632 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.561054 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.561036 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.561509 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.561472 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03208ada-d175-4176-932a-901db8659f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.562083 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.561717 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.563323 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.563290 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.563430 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.563370 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.563640 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.563615 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.563986 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.563954 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03208ada-d175-4176-932a-901db8659f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.564197 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.564171 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.564300 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.564240 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.564401 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.564385 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.564815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.564794 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03208ada-d175-4176-932a-901db8659f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.564873 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.564816 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.569359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.569342 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcpmc\" (UniqueName: \"kubernetes.io/projected/03208ada-d175-4176-932a-901db8659f1f-kube-api-access-qcpmc\") pod \"alertmanager-main-0\" (UID: \"03208ada-d175-4176-932a-901db8659f1f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.661286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661254 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661290 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661310 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661336 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661353 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttdq\" (UniqueName: \"kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661399 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.661517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.661418 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.712716 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.712689 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:21:40.764281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764162 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.764281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764231 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.764724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764438 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.764724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764604 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.764724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764644 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttdq\" (UniqueName: \"kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.765331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764796 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.765331 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.764935 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.766754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.766713 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.770434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.767662 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.770434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.768220 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.770434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.768821 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.782597 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.772082 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.782597 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.772710 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.786014 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.785962 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttdq\" (UniqueName: \"kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq\") pod \"console-7cf6567c9f-vk7qm\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.835990 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.835952 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:40.867635 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.867598 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:21:40.870960 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:21:40.870925 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03208ada_d175_4176_932a_901db8659f1f.slice/crio-bb5cc4aed202e54617a280941c8bd828718da261fdb26516aa40e4e4b5bb7e07 WatchSource:0}: Error finding container bb5cc4aed202e54617a280941c8bd828718da261fdb26516aa40e4e4b5bb7e07: Status 404 returned error can't find the container with id bb5cc4aed202e54617a280941c8bd828718da261fdb26516aa40e4e4b5bb7e07 Jul 10 00:21:40.974563 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:40.974533 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:21:41.011294 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:21:41.011264 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa32af2d_dc72_4910_a514_2d5e5d37a9af.slice/crio-ac0fcfa9e0dbc025844d2d97ecdd4c804e878a765f51b71f461a5216c764725a WatchSource:0}: Error finding container ac0fcfa9e0dbc025844d2d97ecdd4c804e878a765f51b71f461a5216c764725a: Status 404 returned error can't find the container with id ac0fcfa9e0dbc025844d2d97ecdd4c804e878a765f51b71f461a5216c764725a Jul 10 00:21:41.354233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.354134 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf6567c9f-vk7qm" event={"ID":"fa32af2d-dc72-4910-a514-2d5e5d37a9af","Type":"ContainerStarted","Data":"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c"} Jul 10 00:21:41.354233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.354197 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf6567c9f-vk7qm" event={"ID":"fa32af2d-dc72-4910-a514-2d5e5d37a9af","Type":"ContainerStarted","Data":"ac0fcfa9e0dbc025844d2d97ecdd4c804e878a765f51b71f461a5216c764725a"} Jul 10 00:21:41.355676 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.355650 2548 generic.go:356] "Generic (PLEG): container finished" podID="03208ada-d175-4176-932a-901db8659f1f" containerID="5576cbedb5977d271a04345c1cd6eb2b55e2ba12da7e0120ec04342f3e02b8f6" exitCode=0 Jul 10 00:21:41.355769 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.355689 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerDied","Data":"5576cbedb5977d271a04345c1cd6eb2b55e2ba12da7e0120ec04342f3e02b8f6"} Jul 10 00:21:41.355769 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.355733 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"bb5cc4aed202e54617a280941c8bd828718da261fdb26516aa40e4e4b5bb7e07"} Jul 10 00:21:41.373352 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:41.373306 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf6567c9f-vk7qm" podStartSLOduration=1.373286806 podStartE2EDuration="1.373286806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:21:41.372102077 +0000 UTC m=+359.851989828" watchObservedRunningTime="2026-07-10 00:21:41.373286806 +0000 UTC m=+359.853174358" Jul 10 00:21:42.157129 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.157095 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1fc6c99-70c0-499d-9356-cc68a645e9ca" path="/var/lib/kubelet/pods/d1fc6c99-70c0-499d-9356-cc68a645e9ca/volumes" Jul 10 00:21:42.364128 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364039 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"7169f391e420e8f80a8e824f1d251d4dd4c0ffcab8df9c705cac5509f90453b8"} Jul 10 00:21:42.364128 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364122 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"b7976fc127e7eb3a2b9fbe77e98f9c67bd2d2ac50385776ace37a5d2f84de817"} Jul 10 00:21:42.364519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364162 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"574ce395f12bf0b8a4f08d8e3404f4c2092351f87a8bf37912e77fe0d7c55522"} Jul 10 00:21:42.364519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364201 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"5085ff30638373ea2a81ef09660afdff947f39f3fd7bb70d5e28a16a139989a5"} Jul 10 00:21:42.364519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364237 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"151b1f5da173aa0b1cfc41dbed0a134c52dedb16d73e4bd7e36de0536d70f707"} Jul 10 00:21:42.364519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.364271 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03208ada-d175-4176-932a-901db8659f1f","Type":"ContainerStarted","Data":"896050409d5f558e7c9247384eb3a8f305c0b487bb8bb025608ae63a5fe4e7e3"} Jul 10 00:21:42.394431 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:42.394373 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.377601366 podStartE2EDuration="2.394355534s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.754168ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:21:42.388603301 +0000 UTC m=+360.868490850" watchObservedRunningTime="2026-07-10 00:21:42.394355534 +0000 UTC m=+360.874243081" Jul 10 00:21:50.836563 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:50.836521 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:50.837009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:50.836588 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:50.841515 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:50.841474 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:21:51.401797 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:21:51.401770 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:23:11.100181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.100103 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx"] Jul 10 00:23:11.103030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.103014 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.105722 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.105690 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 00:23:11.105864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.105723 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 00:23:11.106940 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.106921 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bq28k\"" Jul 10 00:23:11.110663 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.110641 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx"] Jul 10 00:23:11.172324 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.172288 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zgx\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-kube-api-access-q2zgx\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.172530 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.172336 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.273042 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.272992 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.273229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.273101 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zgx\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-kube-api-access-q2zgx\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.282155 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.282126 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.282532 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.282508 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zgx\" (UniqueName: \"kubernetes.io/projected/917b62b7-e424-42ed-a192-c11bcd8975da-kube-api-access-q2zgx\") pod \"cert-manager-cainjector-559cdd8c87-wx7gx\" (UID: \"917b62b7-e424-42ed-a192-c11bcd8975da\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.413076 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.412995 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" Jul 10 00:23:11.539038 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.539009 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx"] Jul 10 00:23:11.542006 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:23:11.541975 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod917b62b7_e424_42ed_a192_c11bcd8975da.slice/crio-4d7e2d30a7bb2df815681ef563663a0cac5ed1f6c903e07deadedcc903db1439 WatchSource:0}: Error finding container 4d7e2d30a7bb2df815681ef563663a0cac5ed1f6c903e07deadedcc903db1439: Status 404 returned error can't find the container with id 4d7e2d30a7bb2df815681ef563663a0cac5ed1f6c903e07deadedcc903db1439 Jul 10 00:23:11.543936 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.543920 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:23:11.684133 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:11.684041 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" event={"ID":"917b62b7-e424-42ed-a192-c11bcd8975da","Type":"ContainerStarted","Data":"4d7e2d30a7bb2df815681ef563663a0cac5ed1f6c903e07deadedcc903db1439"} Jul 10 00:23:14.699406 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:14.699366 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" event={"ID":"917b62b7-e424-42ed-a192-c11bcd8975da","Type":"ContainerStarted","Data":"5d4924e98877e6059c193ec9fda9954d858e1c6f78951c41876ec2b31af303b9"} Jul 10 00:23:14.716941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:14.716842 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wx7gx" podStartSLOduration=0.802253114 podStartE2EDuration="3.716820603s" totalImagesPullingTime="2.914567489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:14.714395319 +0000 UTC m=+453.194282868" watchObservedRunningTime="2026-07-10 00:23:14.716820603 +0000 UTC m=+453.196708154" Jul 10 00:23:47.148477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.148439 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp"] Jul 10 00:23:47.151716 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.151696 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.154816 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.154796 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 00:23:47.155325 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.155308 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 00:23:47.155451 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.155319 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 00:23:47.155586 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.155572 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 00:23:47.155745 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.155725 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hdnzg\"" Jul 10 00:23:47.171605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.171574 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp"] Jul 10 00:23:47.266460 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.266418 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.266653 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.266511 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.266653 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.266584 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7c4\" (UniqueName: \"kubernetes.io/projected/10514622-9736-4be7-9c30-c44590b111c0-kube-api-access-9f7c4\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.266653 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.266614 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.266653 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.266637 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368025 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368089 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7c4\" (UniqueName: \"kubernetes.io/projected/10514622-9736-4be7-9c30-c44590b111c0-kube-api-access-9f7c4\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368111 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368137 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368157 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.368605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.368582 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-manifests\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.370360 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.370331 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10514622-9736-4be7-9c30-c44590b111c0-tmp\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.370698 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.370676 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.370760 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.370689 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/10514622-9736-4be7-9c30-c44590b111c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.393615 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.393593 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7c4\" (UniqueName: \"kubernetes.io/projected/10514622-9736-4be7-9c30-c44590b111c0-kube-api-access-9f7c4\") pod \"opendatahub-operator-controller-manager-559fc4999-bfhdp\" (UID: \"10514622-9736-4be7-9c30-c44590b111c0\") " pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.462637 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.462605 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:47.623602 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.623577 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp"] Jul 10 00:23:47.626725 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:23:47.626697 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10514622_9736_4be7_9c30_c44590b111c0.slice/crio-d16e3384e6a034088d23147908ee4f3a7603487384b4b725d3f0e58224c57261 WatchSource:0}: Error finding container d16e3384e6a034088d23147908ee4f3a7603487384b4b725d3f0e58224c57261: Status 404 returned error can't find the container with id d16e3384e6a034088d23147908ee4f3a7603487384b4b725d3f0e58224c57261 Jul 10 00:23:47.824991 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:47.824922 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" event={"ID":"10514622-9736-4be7-9c30-c44590b111c0","Type":"ContainerStarted","Data":"d16e3384e6a034088d23147908ee4f3a7603487384b4b725d3f0e58224c57261"} Jul 10 00:23:50.841082 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:50.841042 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" event={"ID":"10514622-9736-4be7-9c30-c44590b111c0","Type":"ContainerStarted","Data":"8559e9c8dcbd9a6b07d8bf6831bad25b67c173e2c99f8df4932a7ed719386f18"} Jul 10 00:23:52.852385 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:52.852347 2548 generic.go:356] "Generic (PLEG): container finished" podID="10514622-9736-4be7-9c30-c44590b111c0" containerID="8559e9c8dcbd9a6b07d8bf6831bad25b67c173e2c99f8df4932a7ed719386f18" exitCode=0 Jul 10 00:23:52.852902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:52.852423 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" event={"ID":"10514622-9736-4be7-9c30-c44590b111c0","Type":"ContainerDied","Data":"8559e9c8dcbd9a6b07d8bf6831bad25b67c173e2c99f8df4932a7ed719386f18"} Jul 10 00:23:53.858214 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:53.858175 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" event={"ID":"10514622-9736-4be7-9c30-c44590b111c0","Type":"ContainerStarted","Data":"76947c324d61295cecf222e4c7fb5f6d6bc29c140ba2e24902415fd93d1a8991"} Jul 10 00:23:53.858620 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:53.858403 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:23:53.883959 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:23:53.883903 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" podStartSLOduration=2.396216074 podStartE2EDuration="6.883881859s" totalImagesPullingTime="3.440950517s" totalInitContainerRuntime="1.046715268s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:53.883094564 +0000 UTC m=+492.362982112" watchObservedRunningTime="2026-07-10 00:23:53.883881859 +0000 UTC m=+492.363769408" Jul 10 00:24:04.864023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:04.863992 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-559fc4999-bfhdp" Jul 10 00:24:06.725805 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.725770 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-879s2"] Jul 10 00:24:06.728838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.728819 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.731671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.731651 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 00:24:06.731794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.731651 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 00:24:06.733162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.733144 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-76m8r\"" Jul 10 00:24:06.733231 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.733160 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 00:24:06.733231 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.733151 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 00:24:06.740868 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.740846 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-879s2"] Jul 10 00:24:06.853733 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.853694 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.853733 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.853730 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tmp\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.853952 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.853829 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmw4\" (UniqueName: \"kubernetes.io/projected/1de655c6-c63a-4ee0-acd6-68f404f95f8e-kube-api-access-hlmw4\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.954678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.954637 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmw4\" (UniqueName: \"kubernetes.io/projected/1de655c6-c63a-4ee0-acd6-68f404f95f8e-kube-api-access-hlmw4\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.954848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.954704 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.954848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.954731 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tmp\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.957145 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.957120 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tmp\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.957263 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.957243 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1de655c6-c63a-4ee0-acd6-68f404f95f8e-tls-certs\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:06.963871 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:06.963848 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmw4\" (UniqueName: \"kubernetes.io/projected/1de655c6-c63a-4ee0-acd6-68f404f95f8e-kube-api-access-hlmw4\") pod \"kube-auth-proxy-b9f4c495f-879s2\" (UID: \"1de655c6-c63a-4ee0-acd6-68f404f95f8e\") " pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:07.038593 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.038471 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" Jul 10 00:24:07.054203 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.054182 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-szt7w"] Jul 10 00:24:07.057382 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.057356 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.061022 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.060997 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 00:24:07.061116 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.061024 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-khxmq\"" Jul 10 00:24:07.069873 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.069850 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-szt7w"] Jul 10 00:24:07.156533 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.156487 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.156681 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.156544 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfc8\" (UniqueName: \"kubernetes.io/projected/0e1047a7-937f-4749-8f28-e09a9b13e5b6-kube-api-access-qtfc8\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.172645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.172615 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b9f4c495f-879s2"] Jul 10 00:24:07.175114 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:24:07.175088 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de655c6_c63a_4ee0_acd6_68f404f95f8e.slice/crio-2165f205d199bd32ca28af8a1ca2c7164b4dbfec648f517c5c1377ac68b4c8b7 WatchSource:0}: Error finding container 2165f205d199bd32ca28af8a1ca2c7164b4dbfec648f517c5c1377ac68b4c8b7: Status 404 returned error can't find the container with id 2165f205d199bd32ca28af8a1ca2c7164b4dbfec648f517c5c1377ac68b4c8b7 Jul 10 00:24:07.257463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.257419 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.257650 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.257474 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfc8\" (UniqueName: \"kubernetes.io/projected/0e1047a7-937f-4749-8f28-e09a9b13e5b6-kube-api-access-qtfc8\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.257650 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:07.257561 2548 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 00:24:07.257650 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:07.257640 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert podName:0e1047a7-937f-4749-8f28-e09a9b13e5b6 nodeName:}" failed. No retries permitted until 2026-07-10 00:24:07.757621461 +0000 UTC m=+506.237508987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert") pod "odh-model-controller-7998c67dc4-szt7w" (UID: "0e1047a7-937f-4749-8f28-e09a9b13e5b6") : secret "odh-model-controller-webhook-cert" not found Jul 10 00:24:07.266433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.266414 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfc8\" (UniqueName: \"kubernetes.io/projected/0e1047a7-937f-4749-8f28-e09a9b13e5b6-kube-api-access-qtfc8\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.760956 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.760920 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.763359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.763329 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0e1047a7-937f-4749-8f28-e09a9b13e5b6-cert\") pod \"odh-model-controller-7998c67dc4-szt7w\" (UID: \"0e1047a7-937f-4749-8f28-e09a9b13e5b6\") " pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:07.911838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.911798 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" event={"ID":"1de655c6-c63a-4ee0-acd6-68f404f95f8e","Type":"ContainerStarted","Data":"2165f205d199bd32ca28af8a1ca2c7164b4dbfec648f517c5c1377ac68b4c8b7"} Jul 10 00:24:07.968100 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:07.968064 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:08.036838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.036779 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:08.042143 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.041983 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.045221 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.045075 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 00:24:08.045221 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.045170 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v775r\"" Jul 10 00:24:08.045400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.045087 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:24:08.045400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.045173 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 00:24:08.045400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.045075 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 00:24:08.052671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.051975 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:08.148871 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.148837 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-szt7w"] Jul 10 00:24:08.153075 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:24:08.153050 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1047a7_937f_4749_8f28_e09a9b13e5b6.slice/crio-5989bef33d38485ed736c75d0692b3a02ec06122a8d31bb5a532e729fffb2fcb WatchSource:0}: Error finding container 5989bef33d38485ed736c75d0692b3a02ec06122a8d31bb5a532e729fffb2fcb: Status 404 returned error can't find the container with id 5989bef33d38485ed736c75d0692b3a02ec06122a8d31bb5a532e729fffb2fcb Jul 10 00:24:08.164529 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164504 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164537 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164567 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164738 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164635 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164738 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164701 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164813 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164745 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84fn\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.164854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.164821 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265351 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265315 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265662 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265365 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265662 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265398 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265662 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265425 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265662 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265629 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265949 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265674 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84fn\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.265949 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.265753 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.266164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.266119 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.268370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.268336 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.268511 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.268392 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.268715 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.268695 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.268793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.268726 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.275281 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.275249 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.275555 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.275531 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84fn\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn\") pod \"istiod-openshift-gateway-7574459656-bsgv9\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.355613 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.355513 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:08.482662 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.482605 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:08.484771 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:24:08.484744 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54054172_493a_42db_85f9_f1179667ff6b.slice/crio-c1ff25a52a5f59a909872b685a2aa770fd0240f3868ef52501a64cf42d23b0df WatchSource:0}: Error finding container c1ff25a52a5f59a909872b685a2aa770fd0240f3868ef52501a64cf42d23b0df: Status 404 returned error can't find the container with id c1ff25a52a5f59a909872b685a2aa770fd0240f3868ef52501a64cf42d23b0df Jul 10 00:24:08.917169 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.917130 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" event={"ID":"0e1047a7-937f-4749-8f28-e09a9b13e5b6","Type":"ContainerStarted","Data":"5989bef33d38485ed736c75d0692b3a02ec06122a8d31bb5a532e729fffb2fcb"} Jul 10 00:24:08.918392 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:08.918370 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" event={"ID":"54054172-493a-42db-85f9-f1179667ff6b","Type":"ContainerStarted","Data":"c1ff25a52a5f59a909872b685a2aa770fd0240f3868ef52501a64cf42d23b0df"} Jul 10 00:24:12.922652 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:12.922614 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:24:12.923010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:12.922682 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:24:13.945209 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.945160 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" event={"ID":"0e1047a7-937f-4749-8f28-e09a9b13e5b6","Type":"ContainerStarted","Data":"df0d5d1183f1fd9f6696addd455f8a2be27f5c68683ddc86e3ec8ab21302aaee"} Jul 10 00:24:13.945720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.945266 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:13.947398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.947304 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" event={"ID":"54054172-493a-42db-85f9-f1179667ff6b","Type":"ContainerStarted","Data":"4bf2c32d06eba7bbeb53a447117c67d38e88d71fb0254fe30824728102f1c403"} Jul 10 00:24:13.947398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.947369 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:13.949644 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.949619 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" event={"ID":"1de655c6-c63a-4ee0-acd6-68f404f95f8e","Type":"ContainerStarted","Data":"75ed071303b1b3654711121f878277d659e709763b39aca639a7962dae004b15"} Jul 10 00:24:13.966798 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:13.966743 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" podStartSLOduration=2.191797442 podStartE2EDuration="6.966725663s" totalImagesPullingTime="4.774928221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:13.96527847 +0000 UTC m=+512.445166019" watchObservedRunningTime="2026-07-10 00:24:13.966725663 +0000 UTC m=+512.446613464" Jul 10 00:24:14.056781 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:14.056720 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b9f4c495f-879s2" podStartSLOduration=2.318306915 podStartE2EDuration="8.056697734s" totalImagesPullingTime="5.738390819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:14.055152404 +0000 UTC m=+512.535039954" watchObservedRunningTime="2026-07-10 00:24:14.056697734 +0000 UTC m=+512.536585283" Jul 10 00:24:14.057703 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:14.057667 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" podStartSLOduration=1.622019743 podStartE2EDuration="6.05765432s" totalImagesPullingTime="4.435634577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:14.005107968 +0000 UTC m=+512.484995516" watchObservedRunningTime="2026-07-10 00:24:14.05765432 +0000 UTC m=+512.537541868" Jul 10 00:24:14.958995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:14.958965 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:17.106110 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.106077 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd"] Jul 10 00:24:17.109539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.109520 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.112620 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.112598 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gdm44\"" Jul 10 00:24:17.112791 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.112774 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 00:24:17.120739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.120719 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd"] Jul 10 00:24:17.133673 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.133651 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt"] Jul 10 00:24:17.136884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.136866 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.140269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.140247 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9dlk5\"" Jul 10 00:24:17.140774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.140754 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 00:24:17.143677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.143653 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfnk\" (UniqueName: \"kubernetes.io/projected/0935a234-7300-4876-b8ba-ea0465781d6c-kube-api-access-hxfnk\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.143782 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.143762 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.150426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.150399 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt"] Jul 10 00:24:17.244190 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.244152 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfnk\" (UniqueName: \"kubernetes.io/projected/0935a234-7300-4876-b8ba-ea0465781d6c-kube-api-access-hxfnk\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.244190 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.244195 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.244412 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.244249 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfs7m\" (UniqueName: \"kubernetes.io/projected/f91ee870-b4fc-4a83-8196-a43a7fefae05-kube-api-access-dfs7m\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.244412 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.244269 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.244412 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.244359 2548 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 00:24:17.244412 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.244408 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert podName:0935a234-7300-4876-b8ba-ea0465781d6c nodeName:}" failed. No retries permitted until 2026-07-10 00:24:17.744392446 +0000 UTC m=+516.224279972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert") pod "kserve-controller-manager-d6f4b65b8-ck4wd" (UID: "0935a234-7300-4876-b8ba-ea0465781d6c") : secret "kserve-webhook-server-cert" not found Jul 10 00:24:17.266864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.266832 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfnk\" (UniqueName: \"kubernetes.io/projected/0935a234-7300-4876-b8ba-ea0465781d6c-kube-api-access-hxfnk\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.344974 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.344944 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.345121 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.344992 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfs7m\" (UniqueName: \"kubernetes.io/projected/f91ee870-b4fc-4a83-8196-a43a7fefae05-kube-api-access-dfs7m\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.345121 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.345100 2548 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 00:24:17.345228 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.345201 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert podName:f91ee870-b4fc-4a83-8196-a43a7fefae05 nodeName:}" failed. No retries permitted until 2026-07-10 00:24:17.84518033 +0000 UTC m=+516.325067861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert") pod "llmisvc-controller-manager-777c6bd7d5-xdtpt" (UID: "f91ee870-b4fc-4a83-8196-a43a7fefae05") : secret "llmisvc-webhook-server-cert" not found Jul 10 00:24:17.367190 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.367128 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfs7m\" (UniqueName: \"kubernetes.io/projected/f91ee870-b4fc-4a83-8196-a43a7fefae05-kube-api-access-dfs7m\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.748892 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.748857 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.751328 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.751302 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0935a234-7300-4876-b8ba-ea0465781d6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-ck4wd\" (UID: \"0935a234-7300-4876-b8ba-ea0465781d6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:17.849576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:17.849530 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:17.849750 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.849693 2548 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 00:24:17.849793 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:24:17.849765 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert podName:f91ee870-b4fc-4a83-8196-a43a7fefae05 nodeName:}" failed. No retries permitted until 2026-07-10 00:24:18.849750102 +0000 UTC m=+517.329637628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert") pod "llmisvc-controller-manager-777c6bd7d5-xdtpt" (UID: "f91ee870-b4fc-4a83-8196-a43a7fefae05") : secret "llmisvc-webhook-server-cert" not found Jul 10 00:24:18.019634 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.019538 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:18.161637 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.161503 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd"] Jul 10 00:24:18.164240 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:24:18.164212 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0935a234_7300_4876_b8ba_ea0465781d6c.slice/crio-7553c910139ca24b7ed7c46b325d50fb0bbe7be170410d83ab6e708208018690 WatchSource:0}: Error finding container 7553c910139ca24b7ed7c46b325d50fb0bbe7be170410d83ab6e708208018690: Status 404 returned error can't find the container with id 7553c910139ca24b7ed7c46b325d50fb0bbe7be170410d83ab6e708208018690 Jul 10 00:24:18.843843 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.843813 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:18.844017 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.843980 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" podUID="54054172-493a-42db-85f9-f1179667ff6b" containerName="discovery" containerID="cri-o://4bf2c32d06eba7bbeb53a447117c67d38e88d71fb0254fe30824728102f1c403" gracePeriod=30 Jul 10 00:24:18.858442 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.858408 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:18.860764 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.860737 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f91ee870-b4fc-4a83-8196-a43a7fefae05-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xdtpt\" (UID: \"f91ee870-b4fc-4a83-8196-a43a7fefae05\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:18.947414 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.947373 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:18.973376 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.973346 2548 generic.go:356] "Generic (PLEG): container finished" podID="54054172-493a-42db-85f9-f1179667ff6b" containerID="4bf2c32d06eba7bbeb53a447117c67d38e88d71fb0254fe30824728102f1c403" exitCode=0 Jul 10 00:24:18.973549 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.973415 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" event={"ID":"54054172-493a-42db-85f9-f1179667ff6b","Type":"ContainerDied","Data":"4bf2c32d06eba7bbeb53a447117c67d38e88d71fb0254fe30824728102f1c403"} Jul 10 00:24:18.974920 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:18.974886 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" event={"ID":"0935a234-7300-4876-b8ba-ea0465781d6c","Type":"ContainerStarted","Data":"7553c910139ca24b7ed7c46b325d50fb0bbe7be170410d83ab6e708208018690"} Jul 10 00:24:19.091182 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.091143 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:19.098754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.098657 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt"] Jul 10 00:24:19.108982 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:24:19.108941 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf91ee870_b4fc_4a83_8196_a43a7fefae05.slice/crio-72b33ef9fda874adf8bd5d3ef94a975b55bcfe2281be238adeb008b161259d27 WatchSource:0}: Error finding container 72b33ef9fda874adf8bd5d3ef94a975b55bcfe2281be238adeb008b161259d27: Status 404 returned error can't find the container with id 72b33ef9fda874adf8bd5d3ef94a975b55bcfe2281be238adeb008b161259d27 Jul 10 00:24:19.162169 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162129 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162169 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162178 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162203 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162226 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n84fn\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162289 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162317 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162355 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token\") pod \"54054172-493a-42db-85f9-f1179667ff6b\" (UID: \"54054172-493a-42db-85f9-f1179667ff6b\") " Jul 10 00:24:19.162973 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.162882 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:24:19.165013 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.164962 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:24:19.166213 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.166149 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:24:19.166313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.166247 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:24:19.166386 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.166331 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn" (OuterVolumeSpecName: "kube-api-access-n84fn") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "kube-api-access-n84fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:24:19.166539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.166503 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts" (OuterVolumeSpecName: "cacerts") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:24:19.166763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.166586 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs" (OuterVolumeSpecName: "local-certs") pod "54054172-493a-42db-85f9-f1179667ff6b" (UID: "54054172-493a-42db-85f9-f1179667ff6b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:24:19.263611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263562 2548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-ca-configmap\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263608 2548 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/54054172-493a-42db-85f9-f1179667ff6b-local-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263623 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263636 2548 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-cacerts\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263648 2548 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-csr-dns-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263659 2548 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/54054172-493a-42db-85f9-f1179667ff6b-istio-kubeconfig\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.263845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.263671 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84fn\" (UniqueName: \"kubernetes.io/projected/54054172-493a-42db-85f9-f1179667ff6b-kube-api-access-n84fn\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:24:19.981170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.981141 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" Jul 10 00:24:19.981349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.981139 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9" event={"ID":"54054172-493a-42db-85f9-f1179667ff6b","Type":"ContainerDied","Data":"c1ff25a52a5f59a909872b685a2aa770fd0240f3868ef52501a64cf42d23b0df"} Jul 10 00:24:19.981349 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.981301 2548 scope.go:117] "RemoveContainer" containerID="4bf2c32d06eba7bbeb53a447117c67d38e88d71fb0254fe30824728102f1c403" Jul 10 00:24:19.982587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:19.982425 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" event={"ID":"f91ee870-b4fc-4a83-8196-a43a7fefae05","Type":"ContainerStarted","Data":"72b33ef9fda874adf8bd5d3ef94a975b55bcfe2281be238adeb008b161259d27"} Jul 10 00:24:20.005476 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:20.005444 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:20.008993 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:20.008957 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bsgv9"] Jul 10 00:24:20.158610 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:20.158577 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54054172-493a-42db-85f9-f1179667ff6b" path="/var/lib/kubelet/pods/54054172-493a-42db-85f9-f1179667ff6b/volumes" Jul 10 00:24:21.996047 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:21.996010 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" event={"ID":"0935a234-7300-4876-b8ba-ea0465781d6c","Type":"ContainerStarted","Data":"4da74e84e2bf8b6954beed61cdf5d39be28a82e1517860d1b517da01a7d27f65"} Jul 10 00:24:21.996439 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:21.996140 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:22.014465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:22.014411 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" podStartSLOduration=1.9434947299999998 podStartE2EDuration="5.014394302s" totalImagesPullingTime="3.070899572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:22.013072309 +0000 UTC m=+520.492959856" watchObservedRunningTime="2026-07-10 00:24:22.014394302 +0000 UTC m=+520.494281851" Jul 10 00:24:23.003923 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:23.003882 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" event={"ID":"f91ee870-b4fc-4a83-8196-a43a7fefae05","Type":"ContainerStarted","Data":"054cb704ac348412824cda5159e9a06a7915c9d50e9254b73dd16253598f7d82"} Jul 10 00:24:23.023935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:23.023876 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" podStartSLOduration=2.344306828 podStartE2EDuration="6.02386071s" totalImagesPullingTime="3.679553882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:23.022433427 +0000 UTC m=+521.502320976" watchObservedRunningTime="2026-07-10 00:24:23.02386071 +0000 UTC m=+521.503748318" Jul 10 00:24:24.007820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:24.007791 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:24:24.959081 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:24.959046 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-szt7w" Jul 10 00:24:53.009433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:53.009400 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ck4wd" Jul 10 00:24:55.014634 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:24:55.014607 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xdtpt" Jul 10 00:25:08.310364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.310321 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:08.310780 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.310744 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54054172-493a-42db-85f9-f1179667ff6b" containerName="discovery" Jul 10 00:25:08.310780 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.310761 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="54054172-493a-42db-85f9-f1179667ff6b" containerName="discovery" Jul 10 00:25:08.310879 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.310823 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="54054172-493a-42db-85f9-f1179667ff6b" containerName="discovery" Jul 10 00:25:08.312879 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.312856 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:08.315452 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.315429 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:25:08.316679 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.316660 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-k5bnj\"" Jul 10 00:25:08.316791 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.316702 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:25:08.323519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.323462 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:08.385018 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.384975 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpn7\" (UniqueName: \"kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7\") pod \"kuadrant-operator-catalog-pj89c\" (UID: \"95cd026e-9d06-4de0-8f79-615d6cb8c6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:08.485678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.485636 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpn7\" (UniqueName: \"kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7\") pod \"kuadrant-operator-catalog-pj89c\" (UID: \"95cd026e-9d06-4de0-8f79-615d6cb8c6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:08.494956 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.494928 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpn7\" (UniqueName: \"kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7\") pod \"kuadrant-operator-catalog-pj89c\" (UID: \"95cd026e-9d06-4de0-8f79-615d6cb8c6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:08.625102 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.625002 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:08.681131 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.681098 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:08.752889 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.752838 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:08.755723 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:08.755697 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cd026e_9d06_4de0_8f79_615d6cb8c6cb.slice/crio-aca35f010b846fd30a3ce0e5346d0765f7d1aa2099ac7d0817037c40da0e62eb WatchSource:0}: Error finding container aca35f010b846fd30a3ce0e5346d0765f7d1aa2099ac7d0817037c40da0e62eb: Status 404 returned error can't find the container with id aca35f010b846fd30a3ce0e5346d0765f7d1aa2099ac7d0817037c40da0e62eb Jul 10 00:25:08.891576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.891476 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wdxsz"] Jul 10 00:25:08.894543 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.894527 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:08.901843 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.901782 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wdxsz"] Jul 10 00:25:08.989926 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:08.989887 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klf7j\" (UniqueName: \"kubernetes.io/projected/eb9e5721-2292-4990-b0fe-560c698c8de8-kube-api-access-klf7j\") pod \"kuadrant-operator-catalog-wdxsz\" (UID: \"eb9e5721-2292-4990-b0fe-560c698c8de8\") " pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:09.090392 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:09.090362 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klf7j\" (UniqueName: \"kubernetes.io/projected/eb9e5721-2292-4990-b0fe-560c698c8de8-kube-api-access-klf7j\") pod \"kuadrant-operator-catalog-wdxsz\" (UID: \"eb9e5721-2292-4990-b0fe-560c698c8de8\") " pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:09.098315 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:09.098288 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klf7j\" (UniqueName: \"kubernetes.io/projected/eb9e5721-2292-4990-b0fe-560c698c8de8-kube-api-access-klf7j\") pod \"kuadrant-operator-catalog-wdxsz\" (UID: \"eb9e5721-2292-4990-b0fe-560c698c8de8\") " pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:09.204990 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:09.204966 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:09.206165 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:09.206140 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" event={"ID":"95cd026e-9d06-4de0-8f79-615d6cb8c6cb","Type":"ContainerStarted","Data":"aca35f010b846fd30a3ce0e5346d0765f7d1aa2099ac7d0817037c40da0e62eb"} Jul 10 00:25:09.327678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:09.327652 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wdxsz"] Jul 10 00:25:09.329547 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:09.329522 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9e5721_2292_4990_b0fe_560c698c8de8.slice/crio-22c818ab275168889c9275652870d18bb6bf8934fab7d9d4085cf8a5dc52952a WatchSource:0}: Error finding container 22c818ab275168889c9275652870d18bb6bf8934fab7d9d4085cf8a5dc52952a: Status 404 returned error can't find the container with id 22c818ab275168889c9275652870d18bb6bf8934fab7d9d4085cf8a5dc52952a Jul 10 00:25:10.213304 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:10.213249 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" event={"ID":"eb9e5721-2292-4990-b0fe-560c698c8de8","Type":"ContainerStarted","Data":"22c818ab275168889c9275652870d18bb6bf8934fab7d9d4085cf8a5dc52952a"} Jul 10 00:25:12.221678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.221638 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" event={"ID":"95cd026e-9d06-4de0-8f79-615d6cb8c6cb","Type":"ContainerStarted","Data":"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0"} Jul 10 00:25:12.222151 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.221687 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" podUID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" containerName="registry-server" containerID="cri-o://1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0" gracePeriod=2 Jul 10 00:25:12.223150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.223118 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" event={"ID":"eb9e5721-2292-4990-b0fe-560c698c8de8","Type":"ContainerStarted","Data":"a868110c0fbb473e25bdfbf0267b2eadee363be6884f022b422ab55034873b80"} Jul 10 00:25:12.240189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.240146 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" podStartSLOduration=1.782072973 podStartE2EDuration="4.240133183s" totalImagesPullingTime="2.45806021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:12.238791253 +0000 UTC m=+570.718678802" watchObservedRunningTime="2026-07-10 00:25:12.240133183 +0000 UTC m=+570.720020732" Jul 10 00:25:12.252892 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.252848 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" podStartSLOduration=2.360165317 podStartE2EDuration="4.252835014s" totalImagesPullingTime="1.892669697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:12.251790248 +0000 UTC m=+570.731677796" watchObservedRunningTime="2026-07-10 00:25:12.252835014 +0000 UTC m=+570.732722561" Jul 10 00:25:12.463741 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.463716 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:12.521249 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.521176 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zpn7\" (UniqueName: \"kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7\") pod \"95cd026e-9d06-4de0-8f79-615d6cb8c6cb\" (UID: \"95cd026e-9d06-4de0-8f79-615d6cb8c6cb\") " Jul 10 00:25:12.523185 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.523159 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7" (OuterVolumeSpecName: "kube-api-access-5zpn7") pod "95cd026e-9d06-4de0-8f79-615d6cb8c6cb" (UID: "95cd026e-9d06-4de0-8f79-615d6cb8c6cb"). InnerVolumeSpecName "kube-api-access-5zpn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:25:12.622487 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:12.622456 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zpn7\" (UniqueName: \"kubernetes.io/projected/95cd026e-9d06-4de0-8f79-615d6cb8c6cb-kube-api-access-5zpn7\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:25:13.227942 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.227903 2548 generic.go:356] "Generic (PLEG): container finished" podID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" containerID="1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0" exitCode=0 Jul 10 00:25:13.228330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.227953 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" Jul 10 00:25:13.228330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.228057 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" event={"ID":"95cd026e-9d06-4de0-8f79-615d6cb8c6cb","Type":"ContainerDied","Data":"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0"} Jul 10 00:25:13.228330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.228117 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pj89c" event={"ID":"95cd026e-9d06-4de0-8f79-615d6cb8c6cb","Type":"ContainerDied","Data":"aca35f010b846fd30a3ce0e5346d0765f7d1aa2099ac7d0817037c40da0e62eb"} Jul 10 00:25:13.228330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.228148 2548 scope.go:117] "RemoveContainer" containerID="1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0" Jul 10 00:25:13.236909 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.236892 2548 scope.go:117] "RemoveContainer" containerID="1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0" Jul 10 00:25:13.237151 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:25:13.237135 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0\": container with ID starting with 1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0 not found: ID does not exist" containerID="1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0" Jul 10 00:25:13.237195 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.237158 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0"} err="failed to get container status \"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0\": rpc error: code = NotFound desc = could not find container \"1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0\": container with ID starting with 1804b0578e3edb5187d886361889ce71a40e599a620f580ccf5c7db6bf0650a0 not found: ID does not exist" Jul 10 00:25:13.248988 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.248961 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:13.252920 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:13.252894 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pj89c"] Jul 10 00:25:14.157098 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:14.157055 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" path="/var/lib/kubelet/pods/95cd026e-9d06-4de0-8f79-615d6cb8c6cb/volumes" Jul 10 00:25:19.205755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:19.205720 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:19.206189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:19.205784 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:19.228412 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:19.228385 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:19.277116 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:19.277089 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wdxsz" Jul 10 00:25:29.011725 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.011687 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8"] Jul 10 00:25:29.012222 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.012020 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" containerName="registry-server" Jul 10 00:25:29.012222 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.012031 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" containerName="registry-server" Jul 10 00:25:29.012222 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.012097 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="95cd026e-9d06-4de0-8f79-615d6cb8c6cb" containerName="registry-server" Jul 10 00:25:29.015131 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.015113 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:29.018911 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.018865 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 00:25:29.019065 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.018967 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mnld9\"" Jul 10 00:25:29.030755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.030725 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8"] Jul 10 00:25:29.176778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.176730 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdjx7\" (UniqueName: \"kubernetes.io/projected/c1fb7182-0bfe-467f-a2e6-88267a309b90-kube-api-access-mdjx7\") pod \"dns-operator-controller-manager-864dd78c4c-tfsl8\" (UID: \"c1fb7182-0bfe-467f-a2e6-88267a309b90\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:29.278056 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.277971 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdjx7\" (UniqueName: \"kubernetes.io/projected/c1fb7182-0bfe-467f-a2e6-88267a309b90-kube-api-access-mdjx7\") pod \"dns-operator-controller-manager-864dd78c4c-tfsl8\" (UID: \"c1fb7182-0bfe-467f-a2e6-88267a309b90\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:29.315202 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.315166 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdjx7\" (UniqueName: \"kubernetes.io/projected/c1fb7182-0bfe-467f-a2e6-88267a309b90-kube-api-access-mdjx7\") pod \"dns-operator-controller-manager-864dd78c4c-tfsl8\" (UID: \"c1fb7182-0bfe-467f-a2e6-88267a309b90\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:29.324706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.324679 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:29.455911 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:29.455884 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8"] Jul 10 00:25:29.458506 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:29.458461 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fb7182_0bfe_467f_a2e6_88267a309b90.slice/crio-8946f5fab1cc3ad2e7b457801800321777374625041f371892327f7a1a9d5b9e WatchSource:0}: Error finding container 8946f5fab1cc3ad2e7b457801800321777374625041f371892327f7a1a9d5b9e: Status 404 returned error can't find the container with id 8946f5fab1cc3ad2e7b457801800321777374625041f371892327f7a1a9d5b9e Jul 10 00:25:30.308613 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:30.308578 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" event={"ID":"c1fb7182-0bfe-467f-a2e6-88267a309b90","Type":"ContainerStarted","Data":"8946f5fab1cc3ad2e7b457801800321777374625041f371892327f7a1a9d5b9e"} Jul 10 00:25:32.319410 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:32.319371 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" event={"ID":"c1fb7182-0bfe-467f-a2e6-88267a309b90","Type":"ContainerStarted","Data":"066118c45175fc4c01404e79edbb6798ec9b447b1624a6422af6ce5830f0b883"} Jul 10 00:25:32.319816 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:32.319530 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:32.338710 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:32.338661 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" podStartSLOduration=1.8925239 podStartE2EDuration="4.338646866s" totalImagesPullingTime="2.446122966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:32.336070429 +0000 UTC m=+590.815957991" watchObservedRunningTime="2026-07-10 00:25:32.338646866 +0000 UTC m=+590.818534413" Jul 10 00:25:33.266916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.266883 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4"] Jul 10 00:25:33.269202 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.269184 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:33.271685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.271661 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xvtzm\"" Jul 10 00:25:33.279730 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.279707 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4"] Jul 10 00:25:33.415834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.415800 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4hc\" (UniqueName: \"kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc\") pod \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" (UID: \"9e2b5989-cdfb-4276-af77-756ac82e9117\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:33.517226 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.517135 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4hc\" (UniqueName: \"kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc\") pod \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" (UID: \"9e2b5989-cdfb-4276-af77-756ac82e9117\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:33.525126 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.525095 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4hc\" (UniqueName: \"kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc\") pod \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" (UID: \"9e2b5989-cdfb-4276-af77-756ac82e9117\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:33.580370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.580339 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:33.706286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:33.706254 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4"] Jul 10 00:25:33.709612 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:33.709579 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2b5989_cdfb_4276_af77_756ac82e9117.slice/crio-24bda78dffc724a0a4f51b93d9da281cf752d7eb455aa37f463e9a04a237b17e WatchSource:0}: Error finding container 24bda78dffc724a0a4f51b93d9da281cf752d7eb455aa37f463e9a04a237b17e: Status 404 returned error can't find the container with id 24bda78dffc724a0a4f51b93d9da281cf752d7eb455aa37f463e9a04a237b17e Jul 10 00:25:34.328721 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:34.328686 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" event={"ID":"9e2b5989-cdfb-4276-af77-756ac82e9117","Type":"ContainerStarted","Data":"24bda78dffc724a0a4f51b93d9da281cf752d7eb455aa37f463e9a04a237b17e"} Jul 10 00:25:36.341624 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:36.341585 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" event={"ID":"9e2b5989-cdfb-4276-af77-756ac82e9117","Type":"ContainerStarted","Data":"b79cd048892d33bf58595f42afcc8fdb7305150a0549cc3a34714458b2be4baf"} Jul 10 00:25:36.342013 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:36.341693 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:36.360625 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:36.360567 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" podStartSLOduration=1.368739198 podStartE2EDuration="3.360551143s" totalImagesPullingTime="1.991811945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:36.36006232 +0000 UTC m=+594.839949872" watchObservedRunningTime="2026-07-10 00:25:36.360551143 +0000 UTC m=+594.840438691" Jul 10 00:25:37.382363 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.382329 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg"] Jul 10 00:25:37.385440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.385423 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:37.388794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.388773 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x5z6q\"" Jul 10 00:25:37.399145 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.399126 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg"] Jul 10 00:25:37.452148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.452101 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gcgp\" (UniqueName: \"kubernetes.io/projected/2bce3542-f022-4c5e-bad5-08241d8fd951-kube-api-access-9gcgp\") pod \"authorino-operator-75cc8d5cb7-c2rgg\" (UID: \"2bce3542-f022-4c5e-bad5-08241d8fd951\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:37.552942 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.552902 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gcgp\" (UniqueName: \"kubernetes.io/projected/2bce3542-f022-4c5e-bad5-08241d8fd951-kube-api-access-9gcgp\") pod \"authorino-operator-75cc8d5cb7-c2rgg\" (UID: \"2bce3542-f022-4c5e-bad5-08241d8fd951\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:37.563294 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.563257 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gcgp\" (UniqueName: \"kubernetes.io/projected/2bce3542-f022-4c5e-bad5-08241d8fd951-kube-api-access-9gcgp\") pod \"authorino-operator-75cc8d5cb7-c2rgg\" (UID: \"2bce3542-f022-4c5e-bad5-08241d8fd951\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:37.695225 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.695137 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:37.767111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.767078 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78779dd596-frwdc"] Jul 10 00:25:37.771045 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.771024 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.785777 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.785752 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78779dd596-frwdc"] Jul 10 00:25:37.835031 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.834990 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg"] Jul 10 00:25:37.837473 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:37.837447 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bce3542_f022_4c5e_bad5_08241d8fd951.slice/crio-6e9eeef57ab6c6133f471a048f4b472161e729eaab4aacaaed9aee5e6bf72660 WatchSource:0}: Error finding container 6e9eeef57ab6c6133f471a048f4b472161e729eaab4aacaaed9aee5e6bf72660: Status 404 returned error can't find the container with id 6e9eeef57ab6c6133f471a048f4b472161e729eaab4aacaaed9aee5e6bf72660 Jul 10 00:25:37.854699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854678 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-console-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854714 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-oauth-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854745 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-service-ca\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854763 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854821 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-trusted-ca-bundle\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854864 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfznw\" (UniqueName: \"kubernetes.io/projected/573cd238-679c-4793-b69b-79ede849ab12-kube-api-access-nfznw\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.854946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.854935 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-oauth-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955265 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955242 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfznw\" (UniqueName: \"kubernetes.io/projected/573cd238-679c-4793-b69b-79ede849ab12-kube-api-access-nfznw\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955280 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-oauth-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955298 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-console-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955326 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-oauth-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955352 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-service-ca\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955583 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955369 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.955583 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.955396 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-trusted-ca-bundle\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.956054 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.956033 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-oauth-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.956138 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.956106 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-console-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.956240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.956219 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-service-ca\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.956286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.956270 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573cd238-679c-4793-b69b-79ede849ab12-trusted-ca-bundle\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.957743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.957724 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-oauth-config\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.957962 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.957941 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573cd238-679c-4793-b69b-79ede849ab12-console-serving-cert\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:37.981297 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:37.981277 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfznw\" (UniqueName: \"kubernetes.io/projected/573cd238-679c-4793-b69b-79ede849ab12-kube-api-access-nfznw\") pod \"console-78779dd596-frwdc\" (UID: \"573cd238-679c-4793-b69b-79ede849ab12\") " pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:38.080761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.080731 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:38.215286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.215261 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78779dd596-frwdc"] Jul 10 00:25:38.218008 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:38.217979 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573cd238_679c_4793_b69b_79ede849ab12.slice/crio-c211109fcd7b997e2b652d3a2f913b276b4242b2062829c3d3e26f6d630e265f WatchSource:0}: Error finding container c211109fcd7b997e2b652d3a2f913b276b4242b2062829c3d3e26f6d630e265f: Status 404 returned error can't find the container with id c211109fcd7b997e2b652d3a2f913b276b4242b2062829c3d3e26f6d630e265f Jul 10 00:25:38.354552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.354506 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78779dd596-frwdc" event={"ID":"573cd238-679c-4793-b69b-79ede849ab12","Type":"ContainerStarted","Data":"45b6084b1525106b4b92a77d6ebf56389b6f2a6f8fccb00dc1a3de5c706f4116"} Jul 10 00:25:38.354740 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.354570 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78779dd596-frwdc" event={"ID":"573cd238-679c-4793-b69b-79ede849ab12","Type":"ContainerStarted","Data":"c211109fcd7b997e2b652d3a2f913b276b4242b2062829c3d3e26f6d630e265f"} Jul 10 00:25:38.356023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.355993 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" event={"ID":"2bce3542-f022-4c5e-bad5-08241d8fd951","Type":"ContainerStarted","Data":"6e9eeef57ab6c6133f471a048f4b472161e729eaab4aacaaed9aee5e6bf72660"} Jul 10 00:25:38.378282 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:38.378227 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78779dd596-frwdc" podStartSLOduration=1.378209357 podStartE2EDuration="1.378209357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:38.376904697 +0000 UTC m=+596.856792245" watchObservedRunningTime="2026-07-10 00:25:38.378209357 +0000 UTC m=+596.858096904" Jul 10 00:25:40.367425 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:40.367391 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" event={"ID":"2bce3542-f022-4c5e-bad5-08241d8fd951","Type":"ContainerStarted","Data":"6f155b2dd75dbbaf29f0ec5c5df49818e5173a133c7dea610c54502021d009d6"} Jul 10 00:25:40.367735 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:40.367533 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:40.383927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:40.383884 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" podStartSLOduration=0.947065761 podStartE2EDuration="3.383870385s" totalImagesPullingTime="2.436804624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:40.382755226 +0000 UTC m=+598.862642775" watchObservedRunningTime="2026-07-10 00:25:40.383870385 +0000 UTC m=+598.863757933" Jul 10 00:25:42.031851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.031824 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:25:42.032187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.032044 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:25:42.277982 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.277900 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc"] Jul 10 00:25:42.282862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.282838 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.285465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.285442 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vkrqw\"" Jul 10 00:25:42.291913 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.291892 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.292005 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.291948 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zg4\" (UniqueName: \"kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.301134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.301108 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc"] Jul 10 00:25:42.393009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.392971 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.393223 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.393039 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zg4\" (UniqueName: \"kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.393426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.393403 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.408778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.408749 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zg4\" (UniqueName: \"kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.592556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.592437 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:42.735089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:42.735011 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc"] Jul 10 00:25:42.737658 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:42.737633 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e69be82_ca8c_4218_9869_70701a44669a.slice/crio-985acd84a231968e002066c49a4e4346119766d8fedc927eb805275d8494dac8 WatchSource:0}: Error finding container 985acd84a231968e002066c49a4e4346119766d8fedc927eb805275d8494dac8: Status 404 returned error can't find the container with id 985acd84a231968e002066c49a4e4346119766d8fedc927eb805275d8494dac8 Jul 10 00:25:43.325162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:43.325134 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-tfsl8" Jul 10 00:25:43.382768 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:43.382740 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" event={"ID":"3e69be82-ca8c-4218-9869-70701a44669a","Type":"ContainerStarted","Data":"985acd84a231968e002066c49a4e4346119766d8fedc927eb805275d8494dac8"} Jul 10 00:25:47.349578 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:47.349534 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:25:48.081079 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:48.081050 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:48.081292 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:48.081107 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:48.085599 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:48.085577 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:48.409810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:48.409729 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78779dd596-frwdc" Jul 10 00:25:48.457592 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:48.457559 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:25:50.418543 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:50.418409 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" event={"ID":"3e69be82-ca8c-4218-9869-70701a44669a","Type":"ContainerStarted","Data":"743a6db87e1f59b57b3f3cb420d371a06fd7670e7e169801f4958cc740009bfd"} Jul 10 00:25:50.418997 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:50.418571 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:25:50.441329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:50.441278 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" podStartSLOduration=1.153344904 podStartE2EDuration="8.441258649s" totalImagesPullingTime="7.287913745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:50.438747732 +0000 UTC m=+608.918635279" watchObservedRunningTime="2026-07-10 00:25:50.441258649 +0000 UTC m=+608.921146271" Jul 10 00:25:51.373864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:51.373837 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c2rgg" Jul 10 00:25:52.659842 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.659807 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh"] Jul 10 00:25:52.663589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.663567 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.666259 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.666236 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 00:25:52.666366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.666327 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 00:25:52.666409 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.666361 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vzvcw\"" Jul 10 00:25:52.672700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.672680 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh"] Jul 10 00:25:52.787864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.787820 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.787864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.787874 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8626e72-64c9-42ad-a97c-95ed2502c95d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.788091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.787947 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg95\" (UniqueName: \"kubernetes.io/projected/a8626e72-64c9-42ad-a97c-95ed2502c95d-kube-api-access-kpg95\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.889295 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.889263 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg95\" (UniqueName: \"kubernetes.io/projected/a8626e72-64c9-42ad-a97c-95ed2502c95d-kube-api-access-kpg95\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.889394 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.889328 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.889394 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.889354 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8626e72-64c9-42ad-a97c-95ed2502c95d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.889529 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:25:52.889456 2548 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 10 00:25:52.889568 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:25:52.889537 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert podName:a8626e72-64c9-42ad-a97c-95ed2502c95d nodeName:}" failed. No retries permitted until 2026-07-10 00:25:53.389520318 +0000 UTC m=+611.869407847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-xvpnh" (UID: "a8626e72-64c9-42ad-a97c-95ed2502c95d") : secret "plugin-serving-cert" not found Jul 10 00:25:52.889921 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.889905 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8626e72-64c9-42ad-a97c-95ed2502c95d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:52.897942 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:52.897924 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg95\" (UniqueName: \"kubernetes.io/projected/a8626e72-64c9-42ad-a97c-95ed2502c95d-kube-api-access-kpg95\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:53.392535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:53.392478 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:53.394813 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:53.394783 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8626e72-64c9-42ad-a97c-95ed2502c95d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-xvpnh\" (UID: \"a8626e72-64c9-42ad-a97c-95ed2502c95d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:53.572933 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:53.572888 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" Jul 10 00:25:53.700341 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:53.700310 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh"] Jul 10 00:25:53.702310 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:25:53.702280 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8626e72_64c9_42ad_a97c_95ed2502c95d.slice/crio-8465da59affbd7aa2769cd46885d38bc295aeea3a0586031eb505814841f3ca7 WatchSource:0}: Error finding container 8465da59affbd7aa2769cd46885d38bc295aeea3a0586031eb505814841f3ca7: Status 404 returned error can't find the container with id 8465da59affbd7aa2769cd46885d38bc295aeea3a0586031eb505814841f3ca7 Jul 10 00:25:54.438326 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:25:54.438290 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" event={"ID":"a8626e72-64c9-42ad-a97c-95ed2502c95d","Type":"ContainerStarted","Data":"8465da59affbd7aa2769cd46885d38bc295aeea3a0586031eb505814841f3ca7"} Jul 10 00:26:01.425593 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:01.425562 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:02.518994 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.518958 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68"] Jul 10 00:26:02.522535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.522515 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.538857 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.538833 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68"] Jul 10 00:26:02.580449 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.580409 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.580657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.580473 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.681958 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.681918 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.682136 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.681975 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.682406 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.682378 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.693442 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.693411 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.834583 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.834479 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:02.994570 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:02.994524 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68"] Jul 10 00:26:02.998837 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:26:02.998801 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf4d1107_68d3_46c1_8197_f27ada41ce0a.slice/crio-c435dcfe47ec9d02a6909f4c05e9fcaf293035371cfff230c3651c5e9fbd61fd WatchSource:0}: Error finding container c435dcfe47ec9d02a6909f4c05e9fcaf293035371cfff230c3651c5e9fbd61fd: Status 404 returned error can't find the container with id c435dcfe47ec9d02a6909f4c05e9fcaf293035371cfff230c3651c5e9fbd61fd Jul 10 00:26:03.184053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.183951 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68"] Jul 10 00:26:03.190865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.190832 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68"] Jul 10 00:26:03.204742 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.204711 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc"] Jul 10 00:26:03.204983 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.204954 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" podUID="3e69be82-ca8c-4218-9869-70701a44669a" containerName="manager" containerID="cri-o://743a6db87e1f59b57b3f3cb420d371a06fd7670e7e169801f4958cc740009bfd" gracePeriod=2 Jul 10 00:26:03.207790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.207764 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:03.208213 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.208197 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" containerName="manager" Jul 10 00:26:03.208213 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.208216 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" containerName="manager" Jul 10 00:26:03.208517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.208314 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" containerName="manager" Jul 10 00:26:03.211902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.211880 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.216450 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.216284 2548 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3e69be82-ca8c-4218-9869-70701a44669a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\": pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.217587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.217357 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc"] Jul 10 00:26:03.218534 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.218458 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.223767 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.223559 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:03.233308 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.233268 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:03.234034 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.234000 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e69be82-ca8c-4218-9869-70701a44669a" containerName="manager" Jul 10 00:26:03.234034 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.234028 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e69be82-ca8c-4218-9869-70701a44669a" containerName="manager" Jul 10 00:26:03.234250 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.234150 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e69be82-ca8c-4218-9869-70701a44669a" containerName="manager" Jul 10 00:26:03.237432 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.237403 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4"] Jul 10 00:26:03.237604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.237583 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.237921 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.237695 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" containerName="manager" containerID="cri-o://b79cd048892d33bf58595f42afcc8fdb7305150a0549cc3a34714458b2be4baf" gracePeriod=2 Jul 10 00:26:03.244372 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.244349 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4"] Jul 10 00:26:03.259198 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.259148 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.261202 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.260956 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:03.279896 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.279869 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8"] Jul 10 00:26:03.280602 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.280431 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" containerName="manager" Jul 10 00:26:03.280602 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.280453 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" containerName="manager" Jul 10 00:26:03.280779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.280619 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" containerName="manager" Jul 10 00:26:03.284306 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.284283 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:03.289274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.289151 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxxx\" (UniqueName: \"kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.289274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.289241 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.306352 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.306273 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.308559 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.308534 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8"] Jul 10 00:26:03.315229 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.315105 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.317640 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.317615 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.390090 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390039 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxxx\" (UniqueName: \"kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.390270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390096 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jkf\" (UniqueName: \"kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.390270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390140 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.390270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390194 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.390270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390218 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ktv\" (UniqueName: \"kubernetes.io/projected/25b9d0dd-77df-4c15-9c14-d27ae4212d6d-kube-api-access-h9ktv\") pod \"limitador-operator-controller-manager-859d7b78bd-t9sf8\" (UID: \"25b9d0dd-77df-4c15-9c14-d27ae4212d6d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:03.390630 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.390608 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.402295 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.402265 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxxx\" (UniqueName: \"kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-x8rgq\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.490000 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.489971 2548 generic.go:356] "Generic (PLEG): container finished" podID="9e2b5989-cdfb-4276-af77-756ac82e9117" containerID="b79cd048892d33bf58595f42afcc8fdb7305150a0549cc3a34714458b2be4baf" exitCode=0 Jul 10 00:26:03.490711 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.490686 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.490846 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.490750 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ktv\" (UniqueName: \"kubernetes.io/projected/25b9d0dd-77df-4c15-9c14-d27ae4212d6d-kube-api-access-h9ktv\") pod \"limitador-operator-controller-manager-859d7b78bd-t9sf8\" (UID: \"25b9d0dd-77df-4c15-9c14-d27ae4212d6d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:03.490905 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.490858 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jkf\" (UniqueName: \"kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.491134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.491111 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.492860 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.492585 2548 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" container="manager" Jul 10 00:26:03.494327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.494254 2548 generic.go:356] "Generic (PLEG): container finished" podID="3e69be82-ca8c-4218-9869-70701a44669a" containerID="743a6db87e1f59b57b3f3cb420d371a06fd7670e7e169801f4958cc740009bfd" exitCode=0 Jul 10 00:26:03.494954 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.494928 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.497163 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.497132 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.497355 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.497339 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.499689 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.499661 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:03.500127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.500107 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.501653 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.501611 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jkf\" (UniqueName: \"kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9xfnk\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.501803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.501785 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ktv\" (UniqueName: \"kubernetes.io/projected/25b9d0dd-77df-4c15-9c14-d27ae4212d6d-kube-api-access-h9ktv\") pod \"limitador-operator-controller-manager-859d7b78bd-t9sf8\" (UID: \"25b9d0dd-77df-4c15-9c14-d27ae4212d6d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:03.501882 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.501854 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:03.503856 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.503822 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.505771 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.505747 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.507931 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.507894 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7zfsc\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="3e69be82-ca8c-4218-9869-70701a44669a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:03.509845 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.509817 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4tlk4\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:03.511860 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:03.511829 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:03.591665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.591630 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt4hc\" (UniqueName: \"kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc\") pod \"9e2b5989-cdfb-4276-af77-756ac82e9117\" (UID: \"9e2b5989-cdfb-4276-af77-756ac82e9117\") " Jul 10 00:26:03.591665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.591667 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67zg4\" (UniqueName: \"kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4\") pod \"3e69be82-ca8c-4218-9869-70701a44669a\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " Jul 10 00:26:03.592075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.591705 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume\") pod \"3e69be82-ca8c-4218-9869-70701a44669a\" (UID: \"3e69be82-ca8c-4218-9869-70701a44669a\") " Jul 10 00:26:03.592250 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.592227 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3e69be82-ca8c-4218-9869-70701a44669a" (UID: "3e69be82-ca8c-4218-9869-70701a44669a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:26:03.593812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.593790 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4" (OuterVolumeSpecName: "kube-api-access-67zg4") pod "3e69be82-ca8c-4218-9869-70701a44669a" (UID: "3e69be82-ca8c-4218-9869-70701a44669a"). InnerVolumeSpecName "kube-api-access-67zg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:03.593860 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.593823 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc" (OuterVolumeSpecName: "kube-api-access-gt4hc") pod "9e2b5989-cdfb-4276-af77-756ac82e9117" (UID: "9e2b5989-cdfb-4276-af77-756ac82e9117"). InnerVolumeSpecName "kube-api-access-gt4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:03.612977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.612944 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:03.621616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.621587 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:03.626624 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.626593 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:03.694712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.694239 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4hc\" (UniqueName: \"kubernetes.io/projected/9e2b5989-cdfb-4276-af77-756ac82e9117-kube-api-access-gt4hc\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:03.694712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.694273 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zg4\" (UniqueName: \"kubernetes.io/projected/3e69be82-ca8c-4218-9869-70701a44669a-kube-api-access-67zg4\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:03.694712 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.694289 2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3e69be82-ca8c-4218-9869-70701a44669a-extensions-socket-volume\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:03.798249 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:03.798209 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:03.802229 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:26:03.802197 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36a9c7e_a11c_4f28_bd7c_dee14b1836af.slice/crio-b92f7864b32d51b2da2e80758eed5f14e8b45184fd2f9144f890dce8ba519288 WatchSource:0}: Error finding container b92f7864b32d51b2da2e80758eed5f14e8b45184fd2f9144f890dce8ba519288: Status 404 returned error can't find the container with id b92f7864b32d51b2da2e80758eed5f14e8b45184fd2f9144f890dce8ba519288 Jul 10 00:26:04.036857 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.036814 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8"] Jul 10 00:26:04.046505 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.043018 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:04.158746 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.158710 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e69be82-ca8c-4218-9869-70701a44669a" path="/var/lib/kubelet/pods/3e69be82-ca8c-4218-9869-70701a44669a/volumes" Jul 10 00:26:04.159184 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.159169 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2b5989-cdfb-4276-af77-756ac82e9117" path="/var/lib/kubelet/pods/9e2b5989-cdfb-4276-af77-756ac82e9117/volumes" Jul 10 00:26:04.500728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.500674 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" event={"ID":"25b9d0dd-77df-4c15-9c14-d27ae4212d6d","Type":"ContainerStarted","Data":"a2e4f0d6f70f8795e5c51703b61e5fe95a33fb1be5ff8601d05847fdd75123a6"} Jul 10 00:26:04.500927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.500751 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:04.500927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.500772 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" event={"ID":"25b9d0dd-77df-4c15-9c14-d27ae4212d6d","Type":"ContainerStarted","Data":"19543a6effec643705a3ad3ed707e8158a6dc384a4eeb4faef0e4a828ff631dd"} Jul 10 00:26:04.501945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.501926 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7zfsc" Jul 10 00:26:04.501945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.501937 2548 scope.go:117] "RemoveContainer" containerID="743a6db87e1f59b57b3f3cb420d371a06fd7670e7e169801f4958cc740009bfd" Jul 10 00:26:04.503728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.503613 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" event={"ID":"e36a9c7e-a11c-4f28-bd7c-dee14b1836af","Type":"ContainerStarted","Data":"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b"} Jul 10 00:26:04.503728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.503662 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" event={"ID":"e36a9c7e-a11c-4f28-bd7c-dee14b1836af","Type":"ContainerStarted","Data":"b92f7864b32d51b2da2e80758eed5f14e8b45184fd2f9144f890dce8ba519288"} Jul 10 00:26:04.503728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.503696 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:04.505465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.505438 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" event={"ID":"dd08d895-e638-4668-a6a0-64e47cd5eb47","Type":"ContainerStarted","Data":"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947"} Jul 10 00:26:04.505576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.505528 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:04.505576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.505556 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" event={"ID":"dd08d895-e638-4668-a6a0-64e47cd5eb47","Type":"ContainerStarted","Data":"29c2a0277e9e1047193a04290301bd7cdc54928099e47a8e22d9f3b06bb63bb2"} Jul 10 00:26:04.506789 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.506770 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4tlk4" Jul 10 00:26:04.508601 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.508554 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" containerName="manager" containerID="cri-o://d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b" gracePeriod=2 Jul 10 00:26:04.511810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.511793 2548 scope.go:117] "RemoveContainer" containerID="b79cd048892d33bf58595f42afcc8fdb7305150a0549cc3a34714458b2be4baf" Jul 10 00:26:04.523183 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.523143 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" podStartSLOduration=1.523130474 podStartE2EDuration="1.523130474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:04.521447797 +0000 UTC m=+623.001335346" watchObservedRunningTime="2026-07-10 00:26:04.523130474 +0000 UTC m=+623.003018093" Jul 10 00:26:04.540398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.540348 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" podStartSLOduration=1.54033152 podStartE2EDuration="1.54033152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:04.540061056 +0000 UTC m=+623.019948606" watchObservedRunningTime="2026-07-10 00:26:04.54033152 +0000 UTC m=+623.020219068" Jul 10 00:26:04.558831 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.558780 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" podStartSLOduration=1.558766055 podStartE2EDuration="1.558766055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:04.557652854 +0000 UTC m=+623.037540403" watchObservedRunningTime="2026-07-10 00:26:04.558766055 +0000 UTC m=+623.038653603" Jul 10 00:26:04.767221 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.767196 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:04.769446 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:04.769418 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:04.906208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.906162 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6\") pod \"af4d1107-68d3-46c1-8197-f27ada41ce0a\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " Jul 10 00:26:04.906208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.906209 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume\") pod \"af4d1107-68d3-46c1-8197-f27ada41ce0a\" (UID: \"af4d1107-68d3-46c1-8197-f27ada41ce0a\") " Jul 10 00:26:04.906585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.906547 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "af4d1107-68d3-46c1-8197-f27ada41ce0a" (UID: "af4d1107-68d3-46c1-8197-f27ada41ce0a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:26:04.908695 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:04.908671 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6" (OuterVolumeSpecName: "kube-api-access-47rh6") pod "af4d1107-68d3-46c1-8197-f27ada41ce0a" (UID: "af4d1107-68d3-46c1-8197-f27ada41ce0a"). InnerVolumeSpecName "kube-api-access-47rh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:05.007273 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:05.007189 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47rh6\" (UniqueName: \"kubernetes.io/projected/af4d1107-68d3-46c1-8197-f27ada41ce0a-kube-api-access-47rh6\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:05.007273 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:05.007221 2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/af4d1107-68d3-46c1-8197-f27ada41ce0a-extensions-socket-volume\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:05.517739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:05.517694 2548 generic.go:356] "Generic (PLEG): container finished" podID="af4d1107-68d3-46c1-8197-f27ada41ce0a" containerID="d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b" exitCode=2 Jul 10 00:26:05.517904 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:05.517763 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:05.517904 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:05.517790 2548 scope.go:117] "RemoveContainer" containerID="d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b" Jul 10 00:26:05.520520 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:05.520464 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:05.531254 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:05.531225 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-q7d68\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q7d68" Jul 10 00:26:06.160292 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:06.160250 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4d1107-68d3-46c1-8197-f27ada41ce0a" path="/var/lib/kubelet/pods/af4d1107-68d3-46c1-8197-f27ada41ce0a/volumes" Jul 10 00:26:13.484397 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:13.484358 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cf6567c9f-vk7qm" podUID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" containerName="console" containerID="cri-o://8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c" gracePeriod=15 Jul 10 00:26:15.522138 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:15.522070 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:15.522655 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:15.522217 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:15.522655 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:15.522610 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t9sf8" Jul 10 00:26:15.602177 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:15.602142 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:15.602405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:15.602376 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" podUID="dd08d895-e638-4668-a6a0-64e47cd5eb47" containerName="manager" containerID="cri-o://2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947" gracePeriod=10 Jul 10 00:26:19.135211 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.134861 2548 scope.go:117] "RemoveContainer" containerID="d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b" Jul 10 00:26:19.135474 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:19.135223 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b\": container with ID starting with d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b not found: ID does not exist" containerID="d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b" Jul 10 00:26:19.135474 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.135305 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b"} err="failed to get container status \"d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b\": rpc error: code = NotFound desc = could not find container \"d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b\": container with ID starting with d6ca6261af9d9ddefb120d8b369b84222c9db564abf90ce409e85ef2dadea42b not found: ID does not exist" Jul 10 00:26:19.283488 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.283464 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:26:19.289454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.289434 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:19.341239 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341209 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341275 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341300 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4jkf\" (UniqueName: \"kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf\") pod \"dd08d895-e638-4668-a6a0-64e47cd5eb47\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " Jul 10 00:26:19.341415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341327 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume\") pod \"dd08d895-e638-4668-a6a0-64e47cd5eb47\" (UID: \"dd08d895-e638-4668-a6a0-64e47cd5eb47\") " Jul 10 00:26:19.341607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341463 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wttdq\" (UniqueName: \"kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341547 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341579 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341622 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341654 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert\") pod \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\" (UID: \"fa32af2d-dc72-4910-a514-2d5e5d37a9af\") " Jul 10 00:26:19.341763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341727 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:26:19.341899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341753 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca" (OuterVolumeSpecName: "service-ca") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:26:19.341899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.341757 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dd08d895-e638-4668-a6a0-64e47cd5eb47" (UID: "dd08d895-e638-4668-a6a0-64e47cd5eb47"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:26:19.342083 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.342061 2548 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-oauth-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.342146 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.342088 2548 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-service-ca\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.342146 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.342102 2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd08d895-e638-4668-a6a0-64e47cd5eb47-extensions-socket-volume\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.342146 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.342118 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config" (OuterVolumeSpecName: "console-config") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:26:19.342262 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.342232 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:26:19.343749 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.343723 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq" (OuterVolumeSpecName: "kube-api-access-wttdq") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "kube-api-access-wttdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:19.343911 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.343897 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:26:19.344056 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.344034 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf" (OuterVolumeSpecName: "kube-api-access-k4jkf") pod "dd08d895-e638-4668-a6a0-64e47cd5eb47" (UID: "dd08d895-e638-4668-a6a0-64e47cd5eb47"). InnerVolumeSpecName "kube-api-access-k4jkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:19.344096 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.344078 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fa32af2d-dc72-4910-a514-2d5e5d37a9af" (UID: "fa32af2d-dc72-4910-a514-2d5e5d37a9af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443238 2548 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-oauth-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443268 2548 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443279 2548 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa32af2d-dc72-4910-a514-2d5e5d37a9af-trusted-ca-bundle\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443287 2548 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa32af2d-dc72-4910-a514-2d5e5d37a9af-console-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443296 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jkf\" (UniqueName: \"kubernetes.io/projected/dd08d895-e638-4668-a6a0-64e47cd5eb47-kube-api-access-k4jkf\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.443327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.443304 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wttdq\" (UniqueName: \"kubernetes.io/projected/fa32af2d-dc72-4910-a514-2d5e5d37a9af-kube-api-access-wttdq\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:19.597248 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.597213 2548 generic.go:356] "Generic (PLEG): container finished" podID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" containerID="8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c" exitCode=0 Jul 10 00:26:19.597414 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.597260 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf6567c9f-vk7qm" Jul 10 00:26:19.597414 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.597301 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf6567c9f-vk7qm" event={"ID":"fa32af2d-dc72-4910-a514-2d5e5d37a9af","Type":"ContainerDied","Data":"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c"} Jul 10 00:26:19.597414 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.597365 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf6567c9f-vk7qm" event={"ID":"fa32af2d-dc72-4910-a514-2d5e5d37a9af","Type":"ContainerDied","Data":"ac0fcfa9e0dbc025844d2d97ecdd4c804e878a765f51b71f461a5216c764725a"} Jul 10 00:26:19.597414 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.597396 2548 scope.go:117] "RemoveContainer" containerID="8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c" Jul 10 00:26:19.598685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.598665 2548 generic.go:356] "Generic (PLEG): container finished" podID="dd08d895-e638-4668-a6a0-64e47cd5eb47" containerID="2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947" exitCode=0 Jul 10 00:26:19.598798 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.598698 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" Jul 10 00:26:19.598798 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.598699 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" event={"ID":"dd08d895-e638-4668-a6a0-64e47cd5eb47","Type":"ContainerDied","Data":"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947"} Jul 10 00:26:19.598798 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.598753 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk" event={"ID":"dd08d895-e638-4668-a6a0-64e47cd5eb47","Type":"ContainerDied","Data":"29c2a0277e9e1047193a04290301bd7cdc54928099e47a8e22d9f3b06bb63bb2"} Jul 10 00:26:19.602197 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.602173 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" event={"ID":"a8626e72-64c9-42ad-a97c-95ed2502c95d","Type":"ContainerStarted","Data":"36ff21365e46a4a4219adf846ec53946737c80436c65f5ae08e0df42abc44461"} Jul 10 00:26:19.607228 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.607194 2548 scope.go:117] "RemoveContainer" containerID="8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c" Jul 10 00:26:19.607588 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:19.607561 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c\": container with ID starting with 8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c not found: ID does not exist" containerID="8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c" Jul 10 00:26:19.607678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.607599 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c"} err="failed to get container status \"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c\": rpc error: code = NotFound desc = could not find container \"8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c\": container with ID starting with 8cac402dc930026beb21cdf0b4edced5d16e88142fef5ef0720e0666b87c231c not found: ID does not exist" Jul 10 00:26:19.607678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.607621 2548 scope.go:117] "RemoveContainer" containerID="2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947" Jul 10 00:26:19.615404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.615387 2548 scope.go:117] "RemoveContainer" containerID="2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947" Jul 10 00:26:19.615672 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:19.615652 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947\": container with ID starting with 2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947 not found: ID does not exist" containerID="2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947" Jul 10 00:26:19.615732 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.615679 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947"} err="failed to get container status \"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947\": rpc error: code = NotFound desc = could not find container \"2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947\": container with ID starting with 2dc4fdf058bb7eac3eae6785f2c93c082ffa8b9db960a9dae2427fdc19eeb947 not found: ID does not exist" Jul 10 00:26:19.619993 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.619957 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-xvpnh" podStartSLOduration=2.128337579 podStartE2EDuration="27.619943849s" totalImagesPullingTime="25.49160627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:19.619599734 +0000 UTC m=+638.099487281" watchObservedRunningTime="2026-07-10 00:26:19.619943849 +0000 UTC m=+638.099831400" Jul 10 00:26:19.641092 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.641067 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:19.652852 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.652824 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9xfnk"] Jul 10 00:26:19.662897 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.662867 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:26:19.669095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:19.669060 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cf6567c9f-vk7qm"] Jul 10 00:26:20.158080 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:20.158039 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd08d895-e638-4668-a6a0-64e47cd5eb47" path="/var/lib/kubelet/pods/dd08d895-e638-4668-a6a0-64e47cd5eb47/volumes" Jul 10 00:26:20.158457 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:20.158443 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" path="/var/lib/kubelet/pods/fa32af2d-dc72-4910-a514-2d5e5d37a9af/volumes" Jul 10 00:26:34.383098 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.383058 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:34.383763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.383424 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" podUID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" containerName="manager" containerID="cri-o://ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b" gracePeriod=10 Jul 10 00:26:34.636303 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.636242 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:34.668166 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.668130 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htxxx\" (UniqueName: \"kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx\") pod \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " Jul 10 00:26:34.668353 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.668194 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume\") pod \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\" (UID: \"e36a9c7e-a11c-4f28-bd7c-dee14b1836af\") " Jul 10 00:26:34.668606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.668575 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e36a9c7e-a11c-4f28-bd7c-dee14b1836af" (UID: "e36a9c7e-a11c-4f28-bd7c-dee14b1836af"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:26:34.670347 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.670323 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx" (OuterVolumeSpecName: "kube-api-access-htxxx") pod "e36a9c7e-a11c-4f28-bd7c-dee14b1836af" (UID: "e36a9c7e-a11c-4f28-bd7c-dee14b1836af"). InnerVolumeSpecName "kube-api-access-htxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:26:34.679155 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.679121 2548 generic.go:356] "Generic (PLEG): container finished" podID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" containerID="ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b" exitCode=0 Jul 10 00:26:34.679287 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.679181 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" Jul 10 00:26:34.679287 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.679229 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" event={"ID":"e36a9c7e-a11c-4f28-bd7c-dee14b1836af","Type":"ContainerDied","Data":"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b"} Jul 10 00:26:34.679402 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.679305 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq" event={"ID":"e36a9c7e-a11c-4f28-bd7c-dee14b1836af","Type":"ContainerDied","Data":"b92f7864b32d51b2da2e80758eed5f14e8b45184fd2f9144f890dce8ba519288"} Jul 10 00:26:34.679402 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.679341 2548 scope.go:117] "RemoveContainer" containerID="ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b" Jul 10 00:26:34.688820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.688803 2548 scope.go:117] "RemoveContainer" containerID="ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b" Jul 10 00:26:34.689121 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:26:34.689103 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b\": container with ID starting with ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b not found: ID does not exist" containerID="ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b" Jul 10 00:26:34.689170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.689129 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b"} err="failed to get container status \"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b\": rpc error: code = NotFound desc = could not find container \"ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b\": container with ID starting with ad42a39d43fb70f0ebd2f3d0aaf683c344417e33a7809809ba79ae6b3bfa9c2b not found: ID does not exist" Jul 10 00:26:34.707104 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.707023 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:34.710646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.710620 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-x8rgq"] Jul 10 00:26:34.769745 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.769703 2548 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-extensions-socket-volume\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:34.769745 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:34.769734 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxxx\" (UniqueName: \"kubernetes.io/projected/e36a9c7e-a11c-4f28-bd7c-dee14b1836af-kube-api-access-htxxx\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:26:36.157319 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:36.157283 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" path="/var/lib/kubelet/pods/e36a9c7e-a11c-4f28-bd7c-dee14b1836af/volumes" Jul 10 00:26:54.538210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538167 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538564 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd08d895-e638-4668-a6a0-64e47cd5eb47" containerName="manager" Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538578 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd08d895-e638-4668-a6a0-64e47cd5eb47" containerName="manager" Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538594 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" containerName="manager" Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538599 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" containerName="manager" Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538612 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" containerName="console" Jul 10 00:26:54.538648 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538617 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" containerName="console" Jul 10 00:26:54.538875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538680 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36a9c7e-a11c-4f28-bd7c-dee14b1836af" containerName="manager" Jul 10 00:26:54.538875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538689 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa32af2d-dc72-4910-a514-2d5e5d37a9af" containerName="console" Jul 10 00:26:54.538875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.538696 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd08d895-e638-4668-a6a0-64e47cd5eb47" containerName="manager" Jul 10 00:26:54.598585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.598538 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:26:54.598754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.598701 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.601544 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.601479 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 00:26:54.629332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.629297 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:26:54.747287 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.747247 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brskl\" (UniqueName: \"kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.747461 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.747311 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.848370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.848275 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.848370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.848361 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brskl\" (UniqueName: \"kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.848943 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.848922 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.856067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.856044 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brskl\" (UniqueName: \"kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl\") pod \"limitador-limitador-64c94fd64f-7zwmq\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.911044 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.911010 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:26:54.916842 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.916811 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:26:54.921642 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.921621 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:54.929566 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.929487 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:26:54.954767 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:54.954733 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:26:55.047009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.046981 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:26:55.049827 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.049802 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgj5k\" (UniqueName: \"kubernetes.io/projected/75634734-f19d-4a59-b49f-4483f56c2dd0-kube-api-access-xgj5k\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.049964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.049873 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75634734-f19d-4a59-b49f-4483f56c2dd0-config-file\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.049964 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:26:55.049871 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90d326a_d158_44d5_8677_284a0519667e.slice/crio-646e9ff5a88a3e9f4d678fce0923ff51e49cee5b162b6a4a26d20b982075b035 WatchSource:0}: Error finding container 646e9ff5a88a3e9f4d678fce0923ff51e49cee5b162b6a4a26d20b982075b035: Status 404 returned error can't find the container with id 646e9ff5a88a3e9f4d678fce0923ff51e49cee5b162b6a4a26d20b982075b035 Jul 10 00:26:55.151187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.151099 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgj5k\" (UniqueName: \"kubernetes.io/projected/75634734-f19d-4a59-b49f-4483f56c2dd0-kube-api-access-xgj5k\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.151187 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.151176 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75634734-f19d-4a59-b49f-4483f56c2dd0-config-file\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.151815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.151797 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75634734-f19d-4a59-b49f-4483f56c2dd0-config-file\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.158766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.158745 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgj5k\" (UniqueName: \"kubernetes.io/projected/75634734-f19d-4a59-b49f-4483f56c2dd0-kube-api-access-xgj5k\") pod \"limitador-limitador-54d6d988d5-2sqfl\" (UID: \"75634734-f19d-4a59-b49f-4483f56c2dd0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.245412 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.245368 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:26:55.319401 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.319365 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:26:55.324093 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.324061 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:26:55.326867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.326833 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7kws9\"" Jul 10 00:26:55.329863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.329834 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:26:55.382354 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.382318 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:26:55.386013 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:26:55.385974 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75634734_f19d_4a59_b49f_4483f56c2dd0.slice/crio-bc4b53e1a39de410c79d53e75f14837dc06c0faf09e21098260bce65be98e62b WatchSource:0}: Error finding container bc4b53e1a39de410c79d53e75f14837dc06c0faf09e21098260bce65be98e62b: Status 404 returned error can't find the container with id bc4b53e1a39de410c79d53e75f14837dc06c0faf09e21098260bce65be98e62b Jul 10 00:26:55.453803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.453773 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxsn\" (UniqueName: \"kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn\") pod \"authorino-58cd955675-krhzh\" (UID: \"25fc41af-46d9-4618-a886-d70d0c0bd1aa\") " pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:26:55.554451 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.554412 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxsn\" (UniqueName: \"kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn\") pod \"authorino-58cd955675-krhzh\" (UID: \"25fc41af-46d9-4618-a886-d70d0c0bd1aa\") " pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:26:55.562080 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.562058 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxsn\" (UniqueName: \"kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn\") pod \"authorino-58cd955675-krhzh\" (UID: \"25fc41af-46d9-4618-a886-d70d0c0bd1aa\") " pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:26:55.636733 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.636696 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:26:55.772875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.772706 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:26:55.776105 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:26:55.776070 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25fc41af_46d9_4618_a886_d70d0c0bd1aa.slice/crio-bfe30e4d622eb866863349952b0f6060659b19872d0b758a023491d7e1ecf59d WatchSource:0}: Error finding container bfe30e4d622eb866863349952b0f6060659b19872d0b758a023491d7e1ecf59d: Status 404 returned error can't find the container with id bfe30e4d622eb866863349952b0f6060659b19872d0b758a023491d7e1ecf59d Jul 10 00:26:55.785091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.785049 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" event={"ID":"75634734-f19d-4a59-b49f-4483f56c2dd0","Type":"ContainerStarted","Data":"bc4b53e1a39de410c79d53e75f14837dc06c0faf09e21098260bce65be98e62b"} Jul 10 00:26:55.787360 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:55.787276 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" event={"ID":"a90d326a-d158-44d5-8677-284a0519667e","Type":"ContainerStarted","Data":"646e9ff5a88a3e9f4d678fce0923ff51e49cee5b162b6a4a26d20b982075b035"} Jul 10 00:26:56.797944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:56.797870 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-krhzh" event={"ID":"25fc41af-46d9-4618-a886-d70d0c0bd1aa","Type":"ContainerStarted","Data":"bfe30e4d622eb866863349952b0f6060659b19872d0b758a023491d7e1ecf59d"} Jul 10 00:26:59.584510 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:26:59.584450 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:27:00.822365 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.822325 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" event={"ID":"a90d326a-d158-44d5-8677-284a0519667e","Type":"ContainerStarted","Data":"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545"} Jul 10 00:27:00.822854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.822439 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:27:00.823892 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.823869 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-krhzh" event={"ID":"25fc41af-46d9-4618-a886-d70d0c0bd1aa","Type":"ContainerStarted","Data":"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851"} Jul 10 00:27:00.823985 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.823886 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-krhzh" podUID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" containerName="authorino" containerID="cri-o://e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851" gracePeriod=30 Jul 10 00:27:00.825313 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.825290 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" event={"ID":"75634734-f19d-4a59-b49f-4483f56c2dd0","Type":"ContainerStarted","Data":"0051236f965c1bbbfd856796d4ac9799317b99687e5ba52757728713cf8be844"} Jul 10 00:27:00.825412 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.825337 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:27:00.840690 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.840643 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" podStartSLOduration=1.886264042 podStartE2EDuration="6.840618957s" totalImagesPullingTime="4.954354915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:00.839644631 +0000 UTC m=+679.319532179" watchObservedRunningTime="2026-07-10 00:27:00.840618957 +0000 UTC m=+679.320506504" Jul 10 00:27:00.854474 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.854427 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" podStartSLOduration=2.249198761 podStartE2EDuration="6.854409506s" totalImagesPullingTime="4.605210745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:00.85370544 +0000 UTC m=+679.333592988" watchObservedRunningTime="2026-07-10 00:27:00.854409506 +0000 UTC m=+679.334297053" Jul 10 00:27:00.866872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:00.866738 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-krhzh" podStartSLOduration=1.651024268 podStartE2EDuration="5.866720193s" totalImagesPullingTime="4.215695925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:00.866554959 +0000 UTC m=+679.346442507" watchObservedRunningTime="2026-07-10 00:27:00.866720193 +0000 UTC m=+679.346607743" Jul 10 00:27:01.070393 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.070366 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:27:01.214091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.214060 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgxsn\" (UniqueName: \"kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn\") pod \"25fc41af-46d9-4618-a886-d70d0c0bd1aa\" (UID: \"25fc41af-46d9-4618-a886-d70d0c0bd1aa\") " Jul 10 00:27:01.216093 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.216071 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn" (OuterVolumeSpecName: "kube-api-access-sgxsn") pod "25fc41af-46d9-4618-a886-d70d0c0bd1aa" (UID: "25fc41af-46d9-4618-a886-d70d0c0bd1aa"). InnerVolumeSpecName "kube-api-access-sgxsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:01.314841 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.314795 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgxsn\" (UniqueName: \"kubernetes.io/projected/25fc41af-46d9-4618-a886-d70d0c0bd1aa-kube-api-access-sgxsn\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:01.831442 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.831402 2548 generic.go:356] "Generic (PLEG): container finished" podID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" containerID="e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851" exitCode=0 Jul 10 00:27:01.831876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.831452 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-krhzh" Jul 10 00:27:01.831876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.831448 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-krhzh" event={"ID":"25fc41af-46d9-4618-a886-d70d0c0bd1aa","Type":"ContainerDied","Data":"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851"} Jul 10 00:27:01.831876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.831564 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-krhzh" event={"ID":"25fc41af-46d9-4618-a886-d70d0c0bd1aa","Type":"ContainerDied","Data":"bfe30e4d622eb866863349952b0f6060659b19872d0b758a023491d7e1ecf59d"} Jul 10 00:27:01.831876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.831582 2548 scope.go:117] "RemoveContainer" containerID="e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851" Jul 10 00:27:01.842608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.842277 2548 scope.go:117] "RemoveContainer" containerID="e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851" Jul 10 00:27:01.842757 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:01.842714 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851\": container with ID starting with e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851 not found: ID does not exist" containerID="e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851" Jul 10 00:27:01.842822 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.842769 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851"} err="failed to get container status \"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851\": rpc error: code = NotFound desc = could not find container \"e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851\": container with ID starting with e09b597515a473a1432bd30f903b38352426c37433d272ab9a022ab24e5d6851 not found: ID does not exist" Jul 10 00:27:01.852440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.852417 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:27:01.855927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:01.855908 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-krhzh"] Jul 10 00:27:02.158946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:02.158869 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" path="/var/lib/kubelet/pods/25fc41af-46d9-4618-a886-d70d0c0bd1aa/volumes" Jul 10 00:27:11.832330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:11.832275 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2sqfl" Jul 10 00:27:11.833725 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:11.833700 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:27:11.887649 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:11.887622 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:27:11.887794 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:11.887772 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" podUID="a90d326a-d158-44d5-8677-284a0519667e" containerName="limitador" containerID="cri-o://51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545" gracePeriod=30 Jul 10 00:27:12.440450 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.440426 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:27:12.515195 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.515161 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file\") pod \"a90d326a-d158-44d5-8677-284a0519667e\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " Jul 10 00:27:12.515195 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.515211 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brskl\" (UniqueName: \"kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl\") pod \"a90d326a-d158-44d5-8677-284a0519667e\" (UID: \"a90d326a-d158-44d5-8677-284a0519667e\") " Jul 10 00:27:12.515582 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.515557 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file" (OuterVolumeSpecName: "config-file") pod "a90d326a-d158-44d5-8677-284a0519667e" (UID: "a90d326a-d158-44d5-8677-284a0519667e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:27:12.517324 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.517300 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl" (OuterVolumeSpecName: "kube-api-access-brskl") pod "a90d326a-d158-44d5-8677-284a0519667e" (UID: "a90d326a-d158-44d5-8677-284a0519667e"). InnerVolumeSpecName "kube-api-access-brskl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:12.616002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.615969 2548 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a90d326a-d158-44d5-8677-284a0519667e-config-file\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:12.616002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.616000 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brskl\" (UniqueName: \"kubernetes.io/projected/a90d326a-d158-44d5-8677-284a0519667e-kube-api-access-brskl\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:12.885524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.885407 2548 generic.go:356] "Generic (PLEG): container finished" podID="a90d326a-d158-44d5-8677-284a0519667e" containerID="51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545" exitCode=0 Jul 10 00:27:12.885524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.885454 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" Jul 10 00:27:12.885524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.885462 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" event={"ID":"a90d326a-d158-44d5-8677-284a0519667e","Type":"ContainerDied","Data":"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545"} Jul 10 00:27:12.886009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.885532 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7zwmq" event={"ID":"a90d326a-d158-44d5-8677-284a0519667e","Type":"ContainerDied","Data":"646e9ff5a88a3e9f4d678fce0923ff51e49cee5b162b6a4a26d20b982075b035"} Jul 10 00:27:12.886009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.885555 2548 scope.go:117] "RemoveContainer" containerID="51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545" Jul 10 00:27:12.894273 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.894090 2548 scope.go:117] "RemoveContainer" containerID="51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545" Jul 10 00:27:12.894357 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:12.894337 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545\": container with ID starting with 51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545 not found: ID does not exist" containerID="51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545" Jul 10 00:27:12.894398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.894365 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545"} err="failed to get container status \"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545\": rpc error: code = NotFound desc = could not find container \"51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545\": container with ID starting with 51a8270faf3fffd3dbe218ed1888b713b8017818459d6225a74bf43990316545 not found: ID does not exist" Jul 10 00:27:12.905818 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.905796 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:27:12.909318 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:12.909295 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7zwmq"] Jul 10 00:27:14.157205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:14.157172 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90d326a-d158-44d5-8677-284a0519667e" path="/var/lib/kubelet/pods/a90d326a-d158-44d5-8677-284a0519667e/volumes" Jul 10 00:27:16.918332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918299 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5dh75"] Jul 10 00:27:16.918849 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918829 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90d326a-d158-44d5-8677-284a0519667e" containerName="limitador" Jul 10 00:27:16.918917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918852 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90d326a-d158-44d5-8677-284a0519667e" containerName="limitador" Jul 10 00:27:16.918917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918873 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" containerName="authorino" Jul 10 00:27:16.918917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918882 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" containerName="authorino" Jul 10 00:27:16.919057 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918964 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fc41af-46d9-4618-a886-d70d0c0bd1aa" containerName="authorino" Jul 10 00:27:16.919057 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.918980 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90d326a-d158-44d5-8677-284a0519667e" containerName="limitador" Jul 10 00:27:16.921345 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.921326 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:16.924070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.924031 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 00:27:16.924218 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.924094 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 00:27:16.925258 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.925240 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lr64c\"" Jul 10 00:27:16.925380 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.925322 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 00:27:16.929419 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.929396 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5dh75"] Jul 10 00:27:16.955217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.955189 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fde59454-4851-4a42-a4b0-917c2f075c5e-data\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:16.955335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:16.955249 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8bn\" (UniqueName: \"kubernetes.io/projected/fde59454-4851-4a42-a4b0-917c2f075c5e-kube-api-access-jd8bn\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.055828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.055795 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fde59454-4851-4a42-a4b0-917c2f075c5e-data\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.055971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.055862 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8bn\" (UniqueName: \"kubernetes.io/projected/fde59454-4851-4a42-a4b0-917c2f075c5e-kube-api-access-jd8bn\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.056145 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.056125 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fde59454-4851-4a42-a4b0-917c2f075c5e-data\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.064665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.064646 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8bn\" (UniqueName: \"kubernetes.io/projected/fde59454-4851-4a42-a4b0-917c2f075c5e-kube-api-access-jd8bn\") pod \"postgres-69fb4594cf-5dh75\" (UID: \"fde59454-4851-4a42-a4b0-917c2f075c5e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.233380 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.233338 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:17.366870 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.366832 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5dh75"] Jul 10 00:27:17.369903 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:27:17.369871 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde59454_4851_4a42_a4b0_917c2f075c5e.slice/crio-e061b59534d6d04b70363a1e6c1a744a0653e112097be7a6c171db3503df6ea4 WatchSource:0}: Error finding container e061b59534d6d04b70363a1e6c1a744a0653e112097be7a6c171db3503df6ea4: Status 404 returned error can't find the container with id e061b59534d6d04b70363a1e6c1a744a0653e112097be7a6c171db3503df6ea4 Jul 10 00:27:17.912311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:17.912271 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" event={"ID":"fde59454-4851-4a42-a4b0-917c2f075c5e","Type":"ContainerStarted","Data":"e061b59534d6d04b70363a1e6c1a744a0653e112097be7a6c171db3503df6ea4"} Jul 10 00:27:32.989752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:32.989713 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" event={"ID":"fde59454-4851-4a42-a4b0-917c2f075c5e","Type":"ContainerStarted","Data":"329bc7887e8fdfbe93a3d7ffab8bdf57dbe6a593e43cfaa7f5e7ecde9e21549c"} Jul 10 00:27:32.990204 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:32.989781 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:33.007083 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:33.007025 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" podStartSLOduration=2.26079128 podStartE2EDuration="17.007003498s" totalImagesPullingTime="14.746212218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:33.004462357 +0000 UTC m=+711.484349907" watchObservedRunningTime="2026-07-10 00:27:33.007003498 +0000 UTC m=+711.486891075" Jul 10 00:27:39.024779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.024706 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5dh75" Jul 10 00:27:39.621848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.621808 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnkmr"] Jul 10 00:27:39.625358 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.625337 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:39.627915 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.627891 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7kws9\"" Jul 10 00:27:39.634407 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.634354 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnkmr"] Jul 10 00:27:39.766896 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.766856 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq64x\" (UniqueName: \"kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x\") pod \"authorino-5d4b67c446-tnkmr\" (UID: \"699ea452-4b05-40d4-b1cc-5bbbb85625a7\") " pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:39.791536 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.791502 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnkmr"] Jul 10 00:27:39.791776 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:39.791755 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zq64x], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-tnkmr" podUID="699ea452-4b05-40d4-b1cc-5bbbb85625a7" Jul 10 00:27:39.825972 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.825934 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-74rmp"] Jul 10 00:27:39.829245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.829226 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:39.831897 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.831878 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 00:27:39.836208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.836179 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-74rmp"] Jul 10 00:27:39.867830 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.867793 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq64x\" (UniqueName: \"kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x\") pod \"authorino-5d4b67c446-tnkmr\" (UID: \"699ea452-4b05-40d4-b1cc-5bbbb85625a7\") " pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:39.874300 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.874227 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-74rmp"] Jul 10 00:27:39.874521 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:39.874486 2548 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zvw9m tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-zvw9m tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-74rmp" podUID="d6ac8d8c-4c98-42b4-9c42-149a14ba2a16" Jul 10 00:27:39.877201 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.877180 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq64x\" (UniqueName: \"kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x\") pod \"authorino-5d4b67c446-tnkmr\" (UID: \"699ea452-4b05-40d4-b1cc-5bbbb85625a7\") " pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:39.895147 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.895113 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:27:39.898798 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.898775 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:39.905871 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.905841 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:27:39.969607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.969557 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:39.969809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:39.969668 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvw9m\" (UniqueName: \"kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.020809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.020766 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.020998 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.020767 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:40.027539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.027487 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.030828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.030806 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:40.070982 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.070942 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khm7\" (UniqueName: \"kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.071165 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.070995 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.071165 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.071089 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.071165 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.071145 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvw9m\" (UniqueName: \"kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.073564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.073542 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.078434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.078408 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvw9m\" (UniqueName: \"kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m\") pod \"authorino-76877994-74rmp\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:40.172224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.172141 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert\") pod \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " Jul 10 00:27:40.172224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.172177 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvw9m\" (UniqueName: \"kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m\") pod \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\" (UID: \"d6ac8d8c-4c98-42b4-9c42-149a14ba2a16\") " Jul 10 00:27:40.172224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.172214 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq64x\" (UniqueName: \"kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x\") pod \"699ea452-4b05-40d4-b1cc-5bbbb85625a7\" (UID: \"699ea452-4b05-40d4-b1cc-5bbbb85625a7\") " Jul 10 00:27:40.172514 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.172308 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khm7\" (UniqueName: \"kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.172514 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.172340 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.174448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.174413 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d6ac8d8c-4c98-42b4-9c42-149a14ba2a16" (UID: "d6ac8d8c-4c98-42b4-9c42-149a14ba2a16"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:27:40.174601 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.174467 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m" (OuterVolumeSpecName: "kube-api-access-zvw9m") pod "d6ac8d8c-4c98-42b4-9c42-149a14ba2a16" (UID: "d6ac8d8c-4c98-42b4-9c42-149a14ba2a16"). InnerVolumeSpecName "kube-api-access-zvw9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:40.174601 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.174472 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x" (OuterVolumeSpecName: "kube-api-access-zq64x") pod "699ea452-4b05-40d4-b1cc-5bbbb85625a7" (UID: "699ea452-4b05-40d4-b1cc-5bbbb85625a7"). InnerVolumeSpecName "kube-api-access-zq64x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:40.174815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.174796 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.179799 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.179777 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khm7\" (UniqueName: \"kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7\") pod \"authorino-567ff4b48b-bkdr4\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.209807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.209770 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:27:40.273010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.272932 2548 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-tls-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:40.273010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.272962 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvw9m\" (UniqueName: \"kubernetes.io/projected/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16-kube-api-access-zvw9m\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:40.273010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.272977 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq64x\" (UniqueName: \"kubernetes.io/projected/699ea452-4b05-40d4-b1cc-5bbbb85625a7-kube-api-access-zq64x\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:27:40.341059 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:40.340973 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:27:40.343905 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:27:40.343866 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d37cf7b_658d_497e_8f6d_2f5a65cc9850.slice/crio-e886f3be43f7eac11f2e95a6e5b1d999b0345b61bb83bdd9e30b170632613c19 WatchSource:0}: Error finding container e886f3be43f7eac11f2e95a6e5b1d999b0345b61bb83bdd9e30b170632613c19: Status 404 returned error can't find the container with id e886f3be43f7eac11f2e95a6e5b1d999b0345b61bb83bdd9e30b170632613c19 Jul 10 00:27:41.026755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.026722 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tnkmr" Jul 10 00:27:41.026922 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.026755 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" event={"ID":"7d37cf7b-658d-497e-8f6d-2f5a65cc9850","Type":"ContainerStarted","Data":"75cf3769d2eec4b9767ce3d75b85b81ba894a2e48df8ca221d8b64b5ec743b0f"} Jul 10 00:27:41.026922 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.026795 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" event={"ID":"7d37cf7b-658d-497e-8f6d-2f5a65cc9850","Type":"ContainerStarted","Data":"e886f3be43f7eac11f2e95a6e5b1d999b0345b61bb83bdd9e30b170632613c19"} Jul 10 00:27:41.026922 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.026889 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-74rmp" Jul 10 00:27:41.042796 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.042755 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" podStartSLOduration=1.5714825879999998 podStartE2EDuration="2.042739354s" totalImagesPullingTime="471.256766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:41.041520135 +0000 UTC m=+719.521407678" watchObservedRunningTime="2026-07-10 00:27:41.042739354 +0000 UTC m=+719.522626929" Jul 10 00:27:41.074724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.074629 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnkmr"] Jul 10 00:27:41.080889 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.080856 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tnkmr"] Jul 10 00:27:41.103249 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.103219 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-74rmp"] Jul 10 00:27:41.106867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:41.106839 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-74rmp"] Jul 10 00:27:42.157781 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:42.157748 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699ea452-4b05-40d4-b1cc-5bbbb85625a7" path="/var/lib/kubelet/pods/699ea452-4b05-40d4-b1cc-5bbbb85625a7/volumes" Jul 10 00:27:42.158125 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:42.158005 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ac8d8c-4c98-42b4-9c42-149a14ba2a16" path="/var/lib/kubelet/pods/d6ac8d8c-4c98-42b4-9c42-149a14ba2a16/volumes" Jul 10 00:27:45.508531 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.508471 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:27:45.513651 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.513625 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.516296 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.516272 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:27:45.516464 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.516443 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:27:45.517610 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.517592 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jvcnw\"" Jul 10 00:27:45.521519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.521480 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:27:45.618226 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.618191 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.618226 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.618232 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd577\" (UniqueName: \"kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.682043 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.682003 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:27:45.685296 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.685279 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.699454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.699431 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:27:45.719036 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.719008 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd577\" (UniqueName: \"kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.719194 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.719106 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.719235 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:45.719195 2548 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 00:27:45.719270 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:27:45.719251 2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert podName:543231d3-bc5f-4d37-a0eb-98192f8b2a5d nodeName:}" failed. No retries permitted until 2026-07-10 00:27:46.219236414 +0000 UTC m=+724.699123940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert") pod "maas-controller-565ccbbcd4-7rrfl" (UID: "543231d3-bc5f-4d37-a0eb-98192f8b2a5d") : secret "maas-controller-webhook-cert" not found Jul 10 00:27:45.727486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.727462 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd577\" (UniqueName: \"kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:45.819604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.819521 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkg5n\" (UniqueName: \"kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.819604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.819556 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.920391 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.920356 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkg5n\" (UniqueName: \"kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.920391 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.920390 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.922772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.922749 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.928429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.928409 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkg5n\" (UniqueName: \"kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n\") pod \"maas-controller-5687559b5f-4k8f9\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:45.994693 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:45.994644 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:46.131082 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:46.131007 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:27:46.133411 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:27:46.133387 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44c311b_a17d_463e_85bd_58649580d73a.slice/crio-f8a880a70059e1d6efd205a9d0d6e6bf2ebb21b310b8804002a90ab36f3f716a WatchSource:0}: Error finding container f8a880a70059e1d6efd205a9d0d6e6bf2ebb21b310b8804002a90ab36f3f716a: Status 404 returned error can't find the container with id f8a880a70059e1d6efd205a9d0d6e6bf2ebb21b310b8804002a90ab36f3f716a Jul 10 00:27:46.223447 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:46.223412 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:46.225777 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:46.225748 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") pod \"maas-controller-565ccbbcd4-7rrfl\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:46.425027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:46.424929 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:46.555968 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:46.555818 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:27:46.558736 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:27:46.558707 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543231d3_bc5f_4d37_a0eb_98192f8b2a5d.slice/crio-b225fa524a64f6a3340de2c0bf750229f5c58ed829cfaf8f57bd673a5528f263 WatchSource:0}: Error finding container b225fa524a64f6a3340de2c0bf750229f5c58ed829cfaf8f57bd673a5528f263: Status 404 returned error can't find the container with id b225fa524a64f6a3340de2c0bf750229f5c58ed829cfaf8f57bd673a5528f263 Jul 10 00:27:47.062486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:47.062435 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-4k8f9" event={"ID":"f44c311b-a17d-463e-85bd-58649580d73a","Type":"ContainerStarted","Data":"f8a880a70059e1d6efd205a9d0d6e6bf2ebb21b310b8804002a90ab36f3f716a"} Jul 10 00:27:47.063878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:47.063849 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" event={"ID":"543231d3-bc5f-4d37-a0eb-98192f8b2a5d","Type":"ContainerStarted","Data":"b225fa524a64f6a3340de2c0bf750229f5c58ed829cfaf8f57bd673a5528f263"} Jul 10 00:27:49.077970 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.077931 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-4k8f9" event={"ID":"f44c311b-a17d-463e-85bd-58649580d73a","Type":"ContainerStarted","Data":"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7"} Jul 10 00:27:49.078571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.078031 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:27:49.079754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.079728 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" event={"ID":"543231d3-bc5f-4d37-a0eb-98192f8b2a5d","Type":"ContainerStarted","Data":"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4"} Jul 10 00:27:49.079868 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.079854 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:27:49.095658 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.095597 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5687559b5f-4k8f9" podStartSLOduration=1.588993607 podStartE2EDuration="4.095580482s" totalImagesPullingTime="2.506586875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:49.093672444 +0000 UTC m=+727.573559993" watchObservedRunningTime="2026-07-10 00:27:49.095580482 +0000 UTC m=+727.575468029" Jul 10 00:27:49.112337 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:27:49.112276 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" podStartSLOduration=2.041318175 podStartE2EDuration="4.112259999s" totalImagesPullingTime="2.070941824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:49.11142141 +0000 UTC m=+727.591308958" watchObservedRunningTime="2026-07-10 00:27:49.112259999 +0000 UTC m=+727.592147591" Jul 10 00:28:00.088807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.088770 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:28:00.089184 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.088849 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:28:00.147316 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.147283 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:28:00.147524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.147483 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" podUID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" containerName="manager" containerID="cri-o://20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4" gracePeriod=30 Jul 10 00:28:00.401823 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.401790 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:28:00.431411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.431378 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd577\" (UniqueName: \"kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577\") pod \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " Jul 10 00:28:00.431657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.431421 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") pod \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\" (UID: \"543231d3-bc5f-4d37-a0eb-98192f8b2a5d\") " Jul 10 00:28:00.433641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.433609 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "543231d3-bc5f-4d37-a0eb-98192f8b2a5d" (UID: "543231d3-bc5f-4d37-a0eb-98192f8b2a5d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:28:00.433641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.433631 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577" (OuterVolumeSpecName: "kube-api-access-wd577") pod "543231d3-bc5f-4d37-a0eb-98192f8b2a5d" (UID: "543231d3-bc5f-4d37-a0eb-98192f8b2a5d"). InnerVolumeSpecName "kube-api-access-wd577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:00.532361 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.532314 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd577\" (UniqueName: \"kubernetes.io/projected/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-kube-api-access-wd577\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:28:00.532361 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:00.532353 2548 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/543231d3-bc5f-4d37-a0eb-98192f8b2a5d-webhook-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:28:01.141872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.141839 2548 generic.go:356] "Generic (PLEG): container finished" podID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" containerID="20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4" exitCode=0 Jul 10 00:28:01.142272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.141886 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" Jul 10 00:28:01.142272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.141930 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" event={"ID":"543231d3-bc5f-4d37-a0eb-98192f8b2a5d","Type":"ContainerDied","Data":"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4"} Jul 10 00:28:01.142272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.141984 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-565ccbbcd4-7rrfl" event={"ID":"543231d3-bc5f-4d37-a0eb-98192f8b2a5d","Type":"ContainerDied","Data":"b225fa524a64f6a3340de2c0bf750229f5c58ed829cfaf8f57bd673a5528f263"} Jul 10 00:28:01.142272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.142006 2548 scope.go:117] "RemoveContainer" containerID="20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4" Jul 10 00:28:01.150676 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.150466 2548 scope.go:117] "RemoveContainer" containerID="20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4" Jul 10 00:28:01.150748 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:28:01.150731 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4\": container with ID starting with 20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4 not found: ID does not exist" containerID="20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4" Jul 10 00:28:01.150790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.150757 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4"} err="failed to get container status \"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4\": rpc error: code = NotFound desc = could not find container \"20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4\": container with ID starting with 20e95e61812978041ced4b2185e732ff03fbac0780ce4a492b37073d640730c4 not found: ID does not exist" Jul 10 00:28:01.163117 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.163092 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:28:01.168415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:01.168394 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-565ccbbcd4-7rrfl"] Jul 10 00:28:02.157539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:02.157484 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" path="/var/lib/kubelet/pods/543231d3-bc5f-4d37-a0eb-98192f8b2a5d/volumes" Jul 10 00:28:24.595554 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.595515 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pnsbg"] Jul 10 00:28:24.596078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.595923 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" containerName="manager" Jul 10 00:28:24.596078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.595937 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" containerName="manager" Jul 10 00:28:24.596078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.596028 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="543231d3-bc5f-4d37-a0eb-98192f8b2a5d" containerName="manager" Jul 10 00:28:24.601035 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.601017 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.603806 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.603780 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 00:28:24.603949 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.603834 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5xp79\"" Jul 10 00:28:24.607858 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.607832 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pnsbg"] Jul 10 00:28:24.659967 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.659928 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:28:24.750244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.750200 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03b31b-075b-4e7f-a352-29cef2d1a77c-plugins-config-volume\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.750458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.750276 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkq59\" (UniqueName: \"kubernetes.io/projected/af03b31b-075b-4e7f-a352-29cef2d1a77c-kube-api-access-gkq59\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.851815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.851705 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03b31b-075b-4e7f-a352-29cef2d1a77c-plugins-config-volume\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.851815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.851755 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkq59\" (UniqueName: \"kubernetes.io/projected/af03b31b-075b-4e7f-a352-29cef2d1a77c-kube-api-access-gkq59\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.852418 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.852393 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af03b31b-075b-4e7f-a352-29cef2d1a77c-plugins-config-volume\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.859740 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.859703 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkq59\" (UniqueName: \"kubernetes.io/projected/af03b31b-075b-4e7f-a352-29cef2d1a77c-kube-api-access-gkq59\") pod \"payload-processing-58f86c7798-pnsbg\" (UID: \"af03b31b-075b-4e7f-a352-29cef2d1a77c\") " pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:24.913667 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:24.913636 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:25.054851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:25.054689 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-pnsbg"] Jul 10 00:28:25.057123 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:28:25.057095 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf03b31b_075b_4e7f_a352_29cef2d1a77c.slice/crio-cbd493e87934b9d15a04faa96015380ebaa7839d0746c1757429fa6d1965899a WatchSource:0}: Error finding container cbd493e87934b9d15a04faa96015380ebaa7839d0746c1757429fa6d1965899a: Status 404 returned error can't find the container with id cbd493e87934b9d15a04faa96015380ebaa7839d0746c1757429fa6d1965899a Jul 10 00:28:25.059026 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:25.059008 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:28:25.258169 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:25.258127 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" event={"ID":"af03b31b-075b-4e7f-a352-29cef2d1a77c","Type":"ContainerStarted","Data":"cbd493e87934b9d15a04faa96015380ebaa7839d0746c1757429fa6d1965899a"} Jul 10 00:28:27.270024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:27.269980 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" event={"ID":"af03b31b-075b-4e7f-a352-29cef2d1a77c","Type":"ContainerStarted","Data":"558bd0c948759b5ff06f42d3f3e5244e3f7aa5e087c7d80fdb2a1e23fbea1179"} Jul 10 00:28:27.270541 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:27.270070 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:27.287191 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:27.287140 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" podStartSLOduration=1.853679613 podStartE2EDuration="3.28712475s" totalImagesPullingTime="1.433445137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:27.285472861 +0000 UTC m=+765.765360409" watchObservedRunningTime="2026-07-10 00:28:27.28712475 +0000 UTC m=+765.767012304" Jul 10 00:28:38.276602 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:38.276570 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-pnsbg" Jul 10 00:28:48.692945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.692913 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd"] Jul 10 00:28:48.697848 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.697822 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.701748 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.701719 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nzpmf\"" Jul 10 00:28:48.701878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.701773 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 00:28:48.701878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.701787 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 00:28:48.702009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.701970 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 00:28:48.704834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.704814 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd"] Jul 10 00:28:48.762557 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762513 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcn4p\" (UniqueName: \"kubernetes.io/projected/2f592778-ad5e-4900-859b-b349dc7675c8-kube-api-access-vcn4p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.762742 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762578 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.762742 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762616 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f592778-ad5e-4900-859b-b349dc7675c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.762742 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762675 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.762742 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762735 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.762976 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.762762 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.863730 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863694 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcn4p\" (UniqueName: \"kubernetes.io/projected/2f592778-ad5e-4900-859b-b349dc7675c8-kube-api-access-vcn4p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.863730 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863733 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.863934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863755 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f592778-ad5e-4900-859b-b349dc7675c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.863934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863872 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.864134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863945 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.864134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.863973 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.864245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.864229 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.864334 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.864312 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.864404 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.864330 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.866034 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.866016 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f592778-ad5e-4900-859b-b349dc7675c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.866209 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.866192 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f592778-ad5e-4900-859b-b349dc7675c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:48.871246 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:48.871225 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcn4p\" (UniqueName: \"kubernetes.io/projected/2f592778-ad5e-4900-859b-b349dc7675c8-kube-api-access-vcn4p\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd\" (UID: \"2f592778-ad5e-4900-859b-b349dc7675c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:49.008984 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:49.008952 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:28:49.142015 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:49.141981 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:28:49.145828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:49.145805 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd"] Jul 10 00:28:49.149045 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:28:49.149020 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f592778_ad5e_4900_859b_b349dc7675c8.slice/crio-7e7a10df367a6321d1a770e37fce517583754c503d82b1c0a19f395eefc889a8 WatchSource:0}: Error finding container 7e7a10df367a6321d1a770e37fce517583754c503d82b1c0a19f395eefc889a8: Status 404 returned error can't find the container with id 7e7a10df367a6321d1a770e37fce517583754c503d82b1c0a19f395eefc889a8 Jul 10 00:28:49.383139 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:49.383010 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" event={"ID":"2f592778-ad5e-4900-859b-b349dc7675c8","Type":"ContainerStarted","Data":"7e7a10df367a6321d1a770e37fce517583754c503d82b1c0a19f395eefc889a8"} Jul 10 00:28:57.634198 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:28:57.634155 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:29:06.468706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:06.468657 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" event={"ID":"2f592778-ad5e-4900-859b-b349dc7675c8","Type":"ContainerStarted","Data":"1cde5ed477e3eb149b1d2d701ce9c6e161fce049efe2f4a92b92896bf53b6821"} Jul 10 00:29:06.486706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:06.486644 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" podStartSLOduration=2.174371403 podStartE2EDuration="18.486627363s" totalImagesPullingTime="16.31225596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:06.484103568 +0000 UTC m=+804.963991127" watchObservedRunningTime="2026-07-10 00:29:06.486627363 +0000 UTC m=+804.966514963" Jul 10 00:29:09.010027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:09.009990 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:29:09.010453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:09.010073 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:29:09.023274 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:09.023246 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:29:09.494436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:09.494405 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd" Jul 10 00:29:15.046339 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:15.046298 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:29:24.991343 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:24.991303 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b"] Jul 10 00:29:24.997875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:24.997855 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.000631 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.000600 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 00:29:25.004097 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.004075 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b"] Jul 10 00:29:25.099035 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.098987 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdgt\" (UniqueName: \"kubernetes.io/projected/dc958413-0dbb-413d-8289-575f3066a5bb-kube-api-access-gmdgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.099035 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.099040 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc958413-0dbb-413d-8289-575f3066a5bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.099279 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.099067 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.099279 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.099143 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.099279 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.099205 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.099395 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.099294 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.199771 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199741 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.199950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199800 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdgt\" (UniqueName: \"kubernetes.io/projected/dc958413-0dbb-413d-8289-575f3066a5bb-kube-api-access-gmdgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.199950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199837 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc958413-0dbb-413d-8289-575f3066a5bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.199950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199863 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.199950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199909 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.200168 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.199971 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.200245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.200223 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.200351 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.200330 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.200413 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.200376 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.201998 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.201981 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc958413-0dbb-413d-8289-575f3066a5bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.202348 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.202331 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc958413-0dbb-413d-8289-575f3066a5bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.207770 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.207752 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdgt\" (UniqueName: \"kubernetes.io/projected/dc958413-0dbb-413d-8289-575f3066a5bb-kube-api-access-gmdgt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b\" (UID: \"dc958413-0dbb-413d-8289-575f3066a5bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.309731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.309672 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:25.438063 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.438020 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b"] Jul 10 00:29:25.440212 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:29:25.440181 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc958413_0dbb_413d_8289_575f3066a5bb.slice/crio-65e90dcfa3c5b6301f4bc17fd5376355459e37672a33c00973fea19bb6f60caa WatchSource:0}: Error finding container 65e90dcfa3c5b6301f4bc17fd5376355459e37672a33c00973fea19bb6f60caa: Status 404 returned error can't find the container with id 65e90dcfa3c5b6301f4bc17fd5376355459e37672a33c00973fea19bb6f60caa Jul 10 00:29:25.559660 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:25.559624 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" event={"ID":"dc958413-0dbb-413d-8289-575f3066a5bb","Type":"ContainerStarted","Data":"65e90dcfa3c5b6301f4bc17fd5376355459e37672a33c00973fea19bb6f60caa"} Jul 10 00:29:26.565864 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:26.565825 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" event={"ID":"dc958413-0dbb-413d-8289-575f3066a5bb","Type":"ContainerStarted","Data":"bd11713d853509e830d605eceee021c69a10f45831daaf3ad7093c60302ae2c1"} Jul 10 00:29:26.583843 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:26.583787 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" podStartSLOduration=2.416795959 podStartE2EDuration="2.583768084s" totalImagesPullingTime="166.972125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:26.582473505 +0000 UTC m=+825.062361054" watchObservedRunningTime="2026-07-10 00:29:26.583768084 +0000 UTC m=+825.063655632" Jul 10 00:29:26.738007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:26.737964 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:29:35.310361 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:35.310328 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:35.310876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:35.310382 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:35.322658 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:35.322632 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:35.620308 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:35.620221 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b" Jul 10 00:29:38.445289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:38.445249 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:29:44.944283 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:29:44.944248 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:30:42.062401 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:42.062369 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:30:42.070334 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:42.070296 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:30:46.753240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:46.753153 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:30:46.753709 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:46.753379 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" podUID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" containerName="authorino" containerID="cri-o://75cf3769d2eec4b9767ce3d75b85b81ba894a2e48df8ca221d8b64b5ec743b0f" gracePeriod=30 Jul 10 00:30:46.958039 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:46.958003 2548 generic.go:356] "Generic (PLEG): container finished" podID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" containerID="75cf3769d2eec4b9767ce3d75b85b81ba894a2e48df8ca221d8b64b5ec743b0f" exitCode=0 Jul 10 00:30:46.958194 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:46.958061 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" event={"ID":"7d37cf7b-658d-497e-8f6d-2f5a65cc9850","Type":"ContainerDied","Data":"75cf3769d2eec4b9767ce3d75b85b81ba894a2e48df8ca221d8b64b5ec743b0f"} Jul 10 00:30:46.997077 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:46.997049 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:30:47.143682 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.143600 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert\") pod \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " Jul 10 00:30:47.143682 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.143651 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8khm7\" (UniqueName: \"kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7\") pod \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\" (UID: \"7d37cf7b-658d-497e-8f6d-2f5a65cc9850\") " Jul 10 00:30:47.145651 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.145623 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7" (OuterVolumeSpecName: "kube-api-access-8khm7") pod "7d37cf7b-658d-497e-8f6d-2f5a65cc9850" (UID: "7d37cf7b-658d-497e-8f6d-2f5a65cc9850"). InnerVolumeSpecName "kube-api-access-8khm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:30:47.153836 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.153810 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7d37cf7b-658d-497e-8f6d-2f5a65cc9850" (UID: "7d37cf7b-658d-497e-8f6d-2f5a65cc9850"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:30:47.244904 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.244864 2548 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-tls-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:30:47.244904 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.244901 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8khm7\" (UniqueName: \"kubernetes.io/projected/7d37cf7b-658d-497e-8f6d-2f5a65cc9850-kube-api-access-8khm7\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:30:47.963711 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.963682 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" Jul 10 00:30:47.963711 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.963707 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-567ff4b48b-bkdr4" event={"ID":"7d37cf7b-658d-497e-8f6d-2f5a65cc9850","Type":"ContainerDied","Data":"e886f3be43f7eac11f2e95a6e5b1d999b0345b61bb83bdd9e30b170632613c19"} Jul 10 00:30:47.965379 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.963751 2548 scope.go:117] "RemoveContainer" containerID="75cf3769d2eec4b9767ce3d75b85b81ba894a2e48df8ca221d8b64b5ec743b0f" Jul 10 00:30:47.986697 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.986672 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:30:47.992780 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:47.992758 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-567ff4b48b-bkdr4"] Jul 10 00:30:48.158072 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:30:48.158033 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" path="/var/lib/kubelet/pods/7d37cf7b-658d-497e-8f6d-2f5a65cc9850/volumes" Jul 10 00:31:16.144999 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:16.144964 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:31:26.245095 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:26.245060 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:31:36.742839 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:36.742795 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:31:46.542179 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:46.542142 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:31:57.233788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:57.233752 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:31:59.666381 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.666337 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:31:59.666903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.666766 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5687559b5f-4k8f9" podUID="f44c311b-a17d-463e-85bd-58649580d73a" containerName="manager" containerID="cri-o://181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7" gracePeriod=30 Jul 10 00:31:59.916928 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.916870 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:31:59.984010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.983975 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert\") pod \"f44c311b-a17d-463e-85bd-58649580d73a\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " Jul 10 00:31:59.984178 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.984035 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkg5n\" (UniqueName: \"kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n\") pod \"f44c311b-a17d-463e-85bd-58649580d73a\" (UID: \"f44c311b-a17d-463e-85bd-58649580d73a\") " Jul 10 00:31:59.986033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.986002 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f44c311b-a17d-463e-85bd-58649580d73a" (UID: "f44c311b-a17d-463e-85bd-58649580d73a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:31:59.986181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:31:59.986061 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n" (OuterVolumeSpecName: "kube-api-access-kkg5n") pod "f44c311b-a17d-463e-85bd-58649580d73a" (UID: "f44c311b-a17d-463e-85bd-58649580d73a"). InnerVolumeSpecName "kube-api-access-kkg5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:00.085576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.085528 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkg5n\" (UniqueName: \"kubernetes.io/projected/f44c311b-a17d-463e-85bd-58649580d73a-kube-api-access-kkg5n\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:32:00.085576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.085571 2548 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44c311b-a17d-463e-85bd-58649580d73a-webhook-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:32:00.315357 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.315317 2548 generic.go:356] "Generic (PLEG): container finished" podID="f44c311b-a17d-463e-85bd-58649580d73a" containerID="181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7" exitCode=0 Jul 10 00:32:00.315582 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.315362 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-4k8f9" event={"ID":"f44c311b-a17d-463e-85bd-58649580d73a","Type":"ContainerDied","Data":"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7"} Jul 10 00:32:00.315582 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.315366 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-4k8f9" Jul 10 00:32:00.315582 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.315405 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-4k8f9" event={"ID":"f44c311b-a17d-463e-85bd-58649580d73a","Type":"ContainerDied","Data":"f8a880a70059e1d6efd205a9d0d6e6bf2ebb21b310b8804002a90ab36f3f716a"} Jul 10 00:32:00.315582 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.315428 2548 scope.go:117] "RemoveContainer" containerID="181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7" Jul 10 00:32:00.323940 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.323919 2548 scope.go:117] "RemoveContainer" containerID="181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7" Jul 10 00:32:00.324184 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:32:00.324165 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7\": container with ID starting with 181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7 not found: ID does not exist" containerID="181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7" Jul 10 00:32:00.324232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.324191 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7"} err="failed to get container status \"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7\": rpc error: code = NotFound desc = could not find container \"181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7\": container with ID starting with 181067017d1f5116378786a6f4cebbe1b98ba575259cc450e9c89212b77627a7 not found: ID does not exist" Jul 10 00:32:00.333382 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.333357 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:32:00.338680 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:00.338654 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5687559b5f-4k8f9"] Jul 10 00:32:01.142808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.142766 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5687559b5f-vkflj"] Jul 10 00:32:01.143370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143349 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f44c311b-a17d-463e-85bd-58649580d73a" containerName="manager" Jul 10 00:32:01.143463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143373 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="f44c311b-a17d-463e-85bd-58649580d73a" containerName="manager" Jul 10 00:32:01.143463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143397 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" containerName="authorino" Jul 10 00:32:01.143463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143405 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" containerName="authorino" Jul 10 00:32:01.143638 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143532 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d37cf7b-658d-497e-8f6d-2f5a65cc9850" containerName="authorino" Jul 10 00:32:01.143638 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.143547 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="f44c311b-a17d-463e-85bd-58649580d73a" containerName="manager" Jul 10 00:32:01.147107 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.147077 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.149545 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.149526 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:32:01.149679 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.149527 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jvcnw\"" Jul 10 00:32:01.150829 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.150816 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:32:01.153831 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.153810 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5687559b5f-vkflj"] Jul 10 00:32:01.196969 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.196941 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d92b957f-1338-471f-be8f-c35744d84181-webhook-cert\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.197099 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.196990 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2fd\" (UniqueName: \"kubernetes.io/projected/d92b957f-1338-471f-be8f-c35744d84181-kube-api-access-6t2fd\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.297682 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.297653 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d92b957f-1338-471f-be8f-c35744d84181-webhook-cert\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.297834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.297708 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2fd\" (UniqueName: \"kubernetes.io/projected/d92b957f-1338-471f-be8f-c35744d84181-kube-api-access-6t2fd\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.299903 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.299881 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d92b957f-1338-471f-be8f-c35744d84181-webhook-cert\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.305589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.305564 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2fd\" (UniqueName: \"kubernetes.io/projected/d92b957f-1338-471f-be8f-c35744d84181-kube-api-access-6t2fd\") pod \"maas-controller-5687559b5f-vkflj\" (UID: \"d92b957f-1338-471f-be8f-c35744d84181\") " pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.459047 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.459014 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:01.590055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:01.590024 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5687559b5f-vkflj"] Jul 10 00:32:01.592662 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:32:01.592623 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92b957f_1338_471f_be8f_c35744d84181.slice/crio-6949262b45a829fae60055b38995bcfb82da139bbfee538b6fc6b5f4bad3e06a WatchSource:0}: Error finding container 6949262b45a829fae60055b38995bcfb82da139bbfee538b6fc6b5f4bad3e06a: Status 404 returned error can't find the container with id 6949262b45a829fae60055b38995bcfb82da139bbfee538b6fc6b5f4bad3e06a Jul 10 00:32:02.157974 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:02.157942 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f44c311b-a17d-463e-85bd-58649580d73a" path="/var/lib/kubelet/pods/f44c311b-a17d-463e-85bd-58649580d73a/volumes" Jul 10 00:32:02.327654 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:02.327621 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-vkflj" event={"ID":"d92b957f-1338-471f-be8f-c35744d84181","Type":"ContainerStarted","Data":"035a7e8adf0b04c0127c409f6ff81f90bff7eaf04beb192d1f995a36e10ee677"} Jul 10 00:32:02.327782 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:02.327680 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:02.327782 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:02.327697 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5687559b5f-vkflj" event={"ID":"d92b957f-1338-471f-be8f-c35744d84181","Type":"ContainerStarted","Data":"6949262b45a829fae60055b38995bcfb82da139bbfee538b6fc6b5f4bad3e06a"} Jul 10 00:32:02.344255 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:02.344210 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5687559b5f-vkflj" podStartSLOduration=0.716817212 podStartE2EDuration="1.344195346s" totalImagesPullingTime="627.378134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:02.343366436 +0000 UTC m=+980.823253996" watchObservedRunningTime="2026-07-10 00:32:02.344195346 +0000 UTC m=+980.824082907" Jul 10 00:32:13.338771 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:13.338684 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5687559b5f-vkflj" Jul 10 00:32:29.244610 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:29.244577 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:32:40.338074 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:32:40.338037 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:33:14.546001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:33:14.545963 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:33:24.541819 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:33:24.541778 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:33:34.855678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:33:34.855598 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:33:40.539257 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:33:40.539222 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:33:50.838507 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:33:50.838455 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:34:00.746752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:34:00.746709 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:34:12.343734 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:34:12.343700 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:34:22.245325 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:34:22.245285 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:34:45.140655 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:34:45.140610 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:34:56.940068 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:34:56.940027 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:35:20.137463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:20.137376 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:35:30.442133 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:30.442093 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:35:42.098122 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:42.098090 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:35:42.101834 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:42.101811 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:35:45.741612 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:45.741573 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:35:55.444636 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:35:55.444591 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:36:04.751303 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:36:04.751259 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:36:14.632616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:36:14.632578 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:36:23.446235 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:36:23.446195 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:38:12.636108 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:38:12.636028 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:38:21.635584 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:38:21.635546 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:38:34.741892 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:38:34.741854 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:38:46.539289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:38:46.539254 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:38:56.539766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:38:56.539729 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:08.140023 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:08.139989 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:16.841595 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:16.841550 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:28.239646 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:28.239601 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:37.451269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:37.451178 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:47.637780 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:47.637741 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:39:58.452260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:39:58.452220 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:40:21.088745 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:21.088648 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:40:30.245330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:30.245288 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:40:39.941465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:39.941424 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:40:42.132328 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:42.132298 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:40:42.138717 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:42.138667 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:40:51.836386 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:40:51.836346 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:05.947950 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:05.947868 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:18.043482 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:18.043429 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:26.938066 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:26.938033 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:37.741999 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:37.741961 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:46.946189 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:46.946152 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:41:56.539150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:41:56.539108 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:09.542845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:09.542809 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:18.746527 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:18.746474 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:28.540387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:28.540351 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:37.743896 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:37.743806 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:50.541264 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:50.541228 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:42:59.433932 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:42:59.433885 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:43:11.245365 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:43:11.245324 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:43:22.954710 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:43:22.954672 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:43:32.440576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:43:32.440538 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:43:43.144527 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:43:43.144477 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:00.341270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:00.341232 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:09.941005 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:09.940968 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:17.814450 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.814415 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:17.818706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.818680 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.821746 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.821716 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-14264514-openshift-default-dockercfg-9cjkw\"" Jul 10 00:44:17.821910 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.821750 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:44:17.831136 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.831106 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:17.896212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896165 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896210 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896226 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896342 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896378 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896404 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ps2f\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896448 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896438 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896538 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.896684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.896555 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997080 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997037 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997090 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997126 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ps2f\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997166 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997228 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997250 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997301 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997327 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997354 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997805 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997769 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997799 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997838 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.997941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.997915 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.998271 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.998247 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:17.999862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:17.999826 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:18.000079 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.000063 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:18.005030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.004994 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:18.005179 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.005114 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ps2f\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f\") pod \"e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:18.131610 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.131511 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:18.277466 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.277440 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:18.280151 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:44:18.280116 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0149ee16_5398_44c3_9f12_88a259604cf2.slice/crio-62252f743ad4d2c5823731f0a6ed689968a4022f087a7929a74efa7a49548a43 WatchSource:0}: Error finding container 62252f743ad4d2c5823731f0a6ed689968a4022f087a7929a74efa7a49548a43: Status 404 returned error can't find the container with id 62252f743ad4d2c5823731f0a6ed689968a4022f087a7929a74efa7a49548a43 Jul 10 00:44:18.282104 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.282083 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:44:18.970415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:18.970377 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" event={"ID":"0149ee16-5398-44c3-9f12-88a259604cf2","Type":"ContainerStarted","Data":"62252f743ad4d2c5823731f0a6ed689968a4022f087a7929a74efa7a49548a43"} Jul 10 00:44:20.939764 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:20.939721 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:21.178444 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:21.178395 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:21.178607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:21.178529 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:21.178676 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:21.178604 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:44:21.990965 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:21.990923 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" event={"ID":"0149ee16-5398-44c3-9f12-88a259604cf2","Type":"ContainerStarted","Data":"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06"} Jul 10 00:44:22.011866 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:22.011804 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podStartSLOduration=2.116110861 podStartE2EDuration="5.01178766s" totalImagesPullingTime="2.895676799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:44:22.00981071 +0000 UTC m=+1720.489698274" watchObservedRunningTime="2026-07-10 00:44:22.01178766 +0000 UTC m=+1720.491675207" Jul 10 00:44:22.132200 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:22.132148 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:22.133571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:22.133544 2548 patch_prober.go:28] interesting pod/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 00:44:22.133718 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:22.133594 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 00:44:23.132955 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:23.132912 2548 patch_prober.go:28] interesting pod/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 00:44:23.133908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:23.133872 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 00:44:24.139517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:24.136012 2548 patch_prober.go:28] interesting pod/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 00:44:24.139517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:24.136088 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 00:44:25.133078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:25.133035 2548 patch_prober.go:28] interesting pod/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 00:44:25.133377 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:25.133346 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 00:44:26.132894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:26.132849 2548 patch_prober.go:28] interesting pod/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 00:44:26.133413 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:26.132924 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 00:44:27.136574 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:27.136535 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:27.137069 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:27.136921 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:27.137678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:27.137661 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:30.256877 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:30.256837 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:30.257337 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:30.257027 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" containerID="cri-o://86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06" gracePeriod=30 Jul 10 00:44:30.744224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:30.744180 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:35.517383 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.517351 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:35.570894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570798 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.570894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570842 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.570894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570859 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.570894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570884 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.571240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570905 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.571240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570922 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ps2f\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.571240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570947 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.571240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.570974 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.571240 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.571095 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:35.571509 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.571276 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:44:35.571509 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.571284 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.571509 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.571376 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:35.572037 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.572012 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:35.573809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.573778 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:35.573929 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.573812 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f" (OuterVolumeSpecName: "kube-api-access-2ps2f") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "kube-api-access-2ps2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:35.573929 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.573811 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:44:35.573929 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.573812 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:35.671758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.671714 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket\") pod \"0149ee16-5398-44c3-9f12-88a259604cf2\" (UID: \"0149ee16-5398-44c3-9f12-88a259604cf2\") " Jul 10 00:44:35.671974 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.671960 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0149ee16-5398-44c3-9f12-88a259604cf2-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.671970 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0149ee16-5398-44c3-9f12-88a259604cf2" (UID: "0149ee16-5398-44c3-9f12-88a259604cf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:44:35.672033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.671980 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672033 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.672020 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ps2f\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-kube-api-access-2ps2f\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.672034 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0149ee16-5398-44c3-9f12-88a259604cf2-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.672045 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.672053 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0149ee16-5398-44c3-9f12-88a259604cf2-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.672188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.672060 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:35.773089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:35.773054 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0149ee16-5398-44c3-9f12-88a259604cf2-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:44:36.076969 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.076929 2548 generic.go:356] "Generic (PLEG): container finished" podID="0149ee16-5398-44c3-9f12-88a259604cf2" containerID="86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06" exitCode=0 Jul 10 00:44:36.077152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.076979 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" Jul 10 00:44:36.077152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.076988 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" event={"ID":"0149ee16-5398-44c3-9f12-88a259604cf2","Type":"ContainerDied","Data":"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06"} Jul 10 00:44:36.077152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.077042 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs" event={"ID":"0149ee16-5398-44c3-9f12-88a259604cf2","Type":"ContainerDied","Data":"62252f743ad4d2c5823731f0a6ed689968a4022f087a7929a74efa7a49548a43"} Jul 10 00:44:36.077152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.077072 2548 scope.go:117] "RemoveContainer" containerID="86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06" Jul 10 00:44:36.087398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.087378 2548 scope.go:117] "RemoveContainer" containerID="86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06" Jul 10 00:44:36.087748 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:44:36.087727 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06\": container with ID starting with 86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06 not found: ID does not exist" containerID="86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06" Jul 10 00:44:36.087807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.087758 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06"} err="failed to get container status \"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06\": rpc error: code = NotFound desc = could not find container \"86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06\": container with ID starting with 86291de4d4430a50e4605dfc127758c8b8598ea12afb1918ea0bd9acc04b5e06 not found: ID does not exist" Jul 10 00:44:36.100132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.100097 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:36.103382 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.103348 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-14264514-openshift-default-85b4fbfc5c-8zgjs"] Jul 10 00:44:36.157559 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:36.157518 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" path="/var/lib/kubelet/pods/0149ee16-5398-44c3-9f12-88a259604cf2/volumes" Jul 10 00:44:39.644132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:39.644092 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:44:48.145600 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:44:48.145558 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:45:42.165245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:45:42.165213 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:45:42.173018 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:45:42.172989 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:46:10.047632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:10.047589 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:46:19.335188 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:19.335145 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:46:23.268593 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.268551 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:23.269162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.269141 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" Jul 10 00:46:23.269252 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.269166 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" Jul 10 00:46:23.269308 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.269258 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="0149ee16-5398-44c3-9f12-88a259604cf2" containerName="istio-proxy" Jul 10 00:46:23.272327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.272305 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.275068 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.275046 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:46:23.275196 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.275074 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fa85739d-openshift-default-dockercfg-xwfgs\"" Jul 10 00:46:23.282527 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.282487 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:23.357808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.357769 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.357981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.357823 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.357981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.357939 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.357981 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.357970 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.358089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.357990 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.358089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.358025 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.358089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.358046 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.358089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.358063 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.358214 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.358092 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62skm\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458688 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62skm\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458735 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458768 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458823 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458839 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458854 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.458977 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.458876 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459248 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459047 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459248 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459087 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459248 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459168 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459395 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459316 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459395 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459332 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459427 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.459633 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.459613 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.461809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.461782 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.462162 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.462144 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.468046 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.468017 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.468427 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.468402 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62skm\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm\") pod \"e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.584443 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.584348 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:23.727871 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.727846 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:23.729937 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:46:23.729908 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a22f75_48dc_4582_8735_e1840a8540b0.slice/crio-41ac3f5ab2a05b55995513096b309cbfe1a5b6c54882eb0c0bf1aa8202ade9e4 WatchSource:0}: Error finding container 41ac3f5ab2a05b55995513096b309cbfe1a5b6c54882eb0c0bf1aa8202ade9e4: Status 404 returned error can't find the container with id 41ac3f5ab2a05b55995513096b309cbfe1a5b6c54882eb0c0bf1aa8202ade9e4 Jul 10 00:46:23.732073 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.732037 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:23.732174 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.732119 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:23.732174 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:23.732145 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:24.628385 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:24.628351 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerStarted","Data":"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb"} Jul 10 00:46:24.628856 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:24.628402 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerStarted","Data":"41ac3f5ab2a05b55995513096b309cbfe1a5b6c54882eb0c0bf1aa8202ade9e4"} Jul 10 00:46:24.652467 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:24.652412 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podStartSLOduration=1.652393912 podStartE2EDuration="1.652393912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:46:24.651262118 +0000 UTC m=+1843.131149679" watchObservedRunningTime="2026-07-10 00:46:24.652393912 +0000 UTC m=+1843.132281460" Jul 10 00:46:25.584866 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:25.584823 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:25.590163 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:25.590132 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:25.633809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:25.633776 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:25.634828 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:25.634805 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:31.950220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:31.950181 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:46:40.635058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.635022 2548 patch_prober.go:28] interesting pod/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 00:46:40.635512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.635090 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 00:46:40.715260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.715226 2548 generic.go:356] "Generic (PLEG): container finished" podID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerID="270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb" exitCode=0 Jul 10 00:46:40.715430 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.715302 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerDied","Data":"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb"} Jul 10 00:46:40.715759 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.715744 2548 scope.go:117] "RemoveContainer" containerID="270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb" Jul 10 00:46:40.716832 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.716804 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:40.716899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.716886 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:40.716939 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:40.716923 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:41.723016 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:41.722973 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerStarted","Data":"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11"} Jul 10 00:46:42.584542 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:42.584473 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:42.584844 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:42.584821 2548 patch_prober.go:28] interesting pod/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 00:46:42.584898 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:42.584874 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 00:46:43.584918 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:43.584872 2548 patch_prober.go:28] interesting pod/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 00:46:43.585324 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:43.584934 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 00:46:43.934075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:43.933966 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:46:44.585403 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:44.585356 2548 patch_prober.go:28] interesting pod/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 10 00:46:44.585932 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:44.585423 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 10 00:46:44.819067 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:44.819028 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:44.819316 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:44.819268 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" containerID="cri-o://e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11" gracePeriod=30 Jul 10 00:46:51.480158 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.480127 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:51.530185 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530094 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530185 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530173 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530206 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530232 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62skm\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530267 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530305 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530350 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530373 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530395 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket\") pod \"d9a22f75-48dc-4582-8735-e1840a8540b0\" (UID: \"d9a22f75-48dc-4582-8735-e1840a8540b0\") " Jul 10 00:46:51.530783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530432 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:51.530890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530771 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:51.530890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530881 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.530995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.530903 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.531053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.531010 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:51.531362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.531091 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:46:51.531362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.531193 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:51.532533 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.532476 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:51.532650 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.532527 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:46:51.532841 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.532822 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm" (OuterVolumeSpecName: "kube-api-access-62skm") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "kube-api-access-62skm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:46:51.532899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.532832 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9a22f75-48dc-4582-8735-e1840a8540b0" (UID: "d9a22f75-48dc-4582-8735-e1840a8540b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:46:51.631576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631532 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631571 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631576 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631582 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62skm\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-kube-api-access-62skm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631593 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631602 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a22f75-48dc-4582-8735-e1840a8540b0-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631609 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a22f75-48dc-4582-8735-e1840a8540b0-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.631820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.631617 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a22f75-48dc-4582-8735-e1840a8540b0-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:46:51.782070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.781971 2548 generic.go:356] "Generic (PLEG): container finished" podID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerID="e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11" exitCode=0 Jul 10 00:46:51.782070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.782018 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerDied","Data":"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11"} Jul 10 00:46:51.782070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.782030 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" Jul 10 00:46:51.782070 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.782061 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx" event={"ID":"d9a22f75-48dc-4582-8735-e1840a8540b0","Type":"ContainerDied","Data":"41ac3f5ab2a05b55995513096b309cbfe1a5b6c54882eb0c0bf1aa8202ade9e4"} Jul 10 00:46:51.782405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.782083 2548 scope.go:117] "RemoveContainer" containerID="e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11" Jul 10 00:46:51.791603 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.791584 2548 scope.go:117] "RemoveContainer" containerID="270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb" Jul 10 00:46:51.800072 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.800050 2548 scope.go:117] "RemoveContainer" containerID="e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11" Jul 10 00:46:51.800403 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:46:51.800371 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11\": container with ID starting with e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11 not found: ID does not exist" containerID="e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11" Jul 10 00:46:51.800519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.800406 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11"} err="failed to get container status \"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11\": rpc error: code = NotFound desc = could not find container \"e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11\": container with ID starting with e35d606b3bd39eb9b682e59e25103f4887c9e6c9001a949de83e013eff87ed11 not found: ID does not exist" Jul 10 00:46:51.800519 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.800426 2548 scope.go:117] "RemoveContainer" containerID="270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb" Jul 10 00:46:51.800695 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:46:51.800674 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb\": container with ID starting with 270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb not found: ID does not exist" containerID="270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb" Jul 10 00:46:51.800741 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.800703 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb"} err="failed to get container status \"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb\": rpc error: code = NotFound desc = could not find container \"270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb\": container with ID starting with 270e32e23ee1a4d787453de71dfbc8181c71b609539453d50fb2ff37bc1504eb not found: ID does not exist" Jul 10 00:46:51.805932 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.805903 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:51.809259 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:51.809235 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fa85739d-openshift-default-7c9d66fcdc-zcssx"] Jul 10 00:46:52.158890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:46:52.158807 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" path="/var/lib/kubelet/pods/d9a22f75-48dc-4582-8735-e1840a8540b0/volumes" Jul 10 00:47:04.317771 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.317678 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:47:04.318277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318252 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.318277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318270 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.318400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318290 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.318400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318299 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.318517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318426 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.318517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.318440 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9a22f75-48dc-4582-8735-e1840a8540b0" containerName="istio-proxy" Jul 10 00:47:04.323100 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.323077 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.326004 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.325980 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9a8ef25d-openshift-default-dockercfg-pjdp4\"" Jul 10 00:47:04.326145 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.326001 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:47:04.334173 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.334147 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:47:04.451802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.451759 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.451802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.451800 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452052 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.451820 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452052 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.451902 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fjd\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452052 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.451998 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452052 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.452036 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.452059 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.452074 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.452215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.452093 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552728 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552786 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552807 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552828 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552853 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.552934 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.552887 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553234 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553089 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553234 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553133 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553234 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553168 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fjd\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553408 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553262 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553408 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553282 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553408 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553386 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553612 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553487 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.553651 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.553594 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.555283 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.555252 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.555623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.555601 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.562700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.562678 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fjd\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.562700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.562693 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token\") pod \"e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.635125 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.635043 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:04.780099 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.779955 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:47:04.783258 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:47:04.783223 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ad1fba_e5c3_4629_811b_468f92bfd84c.slice/crio-650528d071da480cadd7e5ad5a3f20ad967f4b1ac96816da9cae1d91885c3dc8 WatchSource:0}: Error finding container 650528d071da480cadd7e5ad5a3f20ad967f4b1ac96816da9cae1d91885c3dc8: Status 404 returned error can't find the container with id 650528d071da480cadd7e5ad5a3f20ad967f4b1ac96816da9cae1d91885c3dc8 Jul 10 00:47:04.785543 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.785508 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:04.785636 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.785579 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:04.785636 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.785627 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:04.849387 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:04.849359 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerStarted","Data":"650528d071da480cadd7e5ad5a3f20ad967f4b1ac96816da9cae1d91885c3dc8"} Jul 10 00:47:05.855652 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:05.855610 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerStarted","Data":"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de"} Jul 10 00:47:05.876869 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:05.876816 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podStartSLOduration=1.876795241 podStartE2EDuration="1.876795241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:47:05.874430352 +0000 UTC m=+1884.354317901" watchObservedRunningTime="2026-07-10 00:47:05.876795241 +0000 UTC m=+1884.356682789" Jul 10 00:47:06.636124 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:06.636086 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:06.641418 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:06.641388 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:06.861045 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:06.861012 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:06.862157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:06.862134 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:14.238674 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:14.238636 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:47:21.375347 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.375311 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:47:21.379508 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.379458 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.382255 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.382227 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9a8ef25d\"/\"openshift-service-ca.crt\"" Jul 10 00:47:21.382504 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.382473 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9a8ef25d\"/\"default-dockercfg-kd5w8\"" Jul 10 00:47:21.383378 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.383355 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9a8ef25d\"/\"e2e-lifecycle-model-9a8ef25d-kserve-self-signed-certs\"" Jul 10 00:47:21.383530 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.383384 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9a8ef25d\"/\"kube-root-ca.crt\"" Jul 10 00:47:21.387075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.387043 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:47:21.522774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522720 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.522774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522785 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.523058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522807 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.523058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522827 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.523058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522858 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttgr\" (UniqueName: \"kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.523058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.522884 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624041 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.623986 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624041 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624042 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624114 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624145 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624187 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttgr\" (UniqueName: \"kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624335 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624207 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624629 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624578 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624647 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.624700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.624684 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.626536 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.626453 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.626917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.626893 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.633289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.633249 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttgr\" (UniqueName: \"kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr\") pod \"e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.694467 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.694420 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:21.837692 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.837659 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:47:21.841291 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:47:21.841255 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18513eca_f568_4cda_bdef_e4c10a5cfd45.slice/crio-8b3c315f97679b4ca1353d915d02dafb8dde7348274e93fdfadcc3450f8b1cd3 WatchSource:0}: Error finding container 8b3c315f97679b4ca1353d915d02dafb8dde7348274e93fdfadcc3450f8b1cd3: Status 404 returned error can't find the container with id 8b3c315f97679b4ca1353d915d02dafb8dde7348274e93fdfadcc3450f8b1cd3 Jul 10 00:47:21.939440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.939392 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" event={"ID":"18513eca-f568-4cda-bdef-e4c10a5cfd45","Type":"ContainerStarted","Data":"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed"} Jul 10 00:47:21.939638 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.939448 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" event={"ID":"18513eca-f568-4cda-bdef-e4c10a5cfd45","Type":"ContainerStarted","Data":"8b3c315f97679b4ca1353d915d02dafb8dde7348274e93fdfadcc3450f8b1cd3"} Jul 10 00:47:21.956328 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:21.956266 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" podStartSLOduration=0.956247145 podStartE2EDuration="956.247145ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:47:21.954319636 +0000 UTC m=+1900.434207187" watchObservedRunningTime="2026-07-10 00:47:21.956247145 +0000 UTC m=+1900.436134692" Jul 10 00:47:22.438329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.438252 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:39770->10.133.0.52:15021: read: connection reset by peer" start-of-body= Jul 10 00:47:22.438788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.438326 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:39770->10.133.0.52:15021: read: connection reset by peer" Jul 10 00:47:22.945545 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.945501 2548 generic.go:356] "Generic (PLEG): container finished" podID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerID="02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de" exitCode=0 Jul 10 00:47:22.945761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.945566 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerDied","Data":"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de"} Jul 10 00:47:22.946137 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.946118 2548 scope.go:117] "RemoveContainer" containerID="02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de" Jul 10 00:47:22.947080 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.947054 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:22.947144 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.947131 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:22.947232 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:22.947167 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:47:23.956787 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:23.956741 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerStarted","Data":"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de"} Jul 10 00:47:24.636347 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:24.636300 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:24.636606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:24.636449 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 00:47:24.636606 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:24.636516 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 00:47:25.636062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:25.636018 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 00:47:25.636743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:25.636709 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 00:47:26.041307 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:26.041258 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:47:26.636383 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:26.636343 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 00:47:26.637431 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:26.637002 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 00:47:27.636159 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:27.636118 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 10 00:47:27.636465 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:27.636435 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 10 00:47:28.639441 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:28.639409 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:28.639884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:28.639802 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:28.640641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:28.640621 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:47:31.694656 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:31.694612 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:31.695065 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:31.694672 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:31.707664 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:31.707631 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:32.008926 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:32.008892 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:47:43.045528 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:43.045474 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:47:56.847678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:56.847640 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:47:56.934467 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:47:56.934428 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:48:11.869148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:11.869101 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerName="main" containerID="cri-o://28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed" gracePeriod=45 Jul 10 00:48:11.999206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:11.999156 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused" Jul 10 00:48:12.132969 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.132893 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:48:12.201046 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.201009 2548 generic.go:356] "Generic (PLEG): container finished" podID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerID="28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed" exitCode=0 Jul 10 00:48:12.201220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.201060 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" Jul 10 00:48:12.201220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.201100 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" event={"ID":"18513eca-f568-4cda-bdef-e4c10a5cfd45","Type":"ContainerDied","Data":"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed"} Jul 10 00:48:12.201220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.201150 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk" event={"ID":"18513eca-f568-4cda-bdef-e4c10a5cfd45","Type":"ContainerDied","Data":"8b3c315f97679b4ca1353d915d02dafb8dde7348274e93fdfadcc3450f8b1cd3"} Jul 10 00:48:12.201220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.201173 2548 scope.go:117] "RemoveContainer" containerID="28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed" Jul 10 00:48:12.210266 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.210245 2548 scope.go:117] "RemoveContainer" containerID="28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed" Jul 10 00:48:12.210547 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:48:12.210528 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed\": container with ID starting with 28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed not found: ID does not exist" containerID="28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed" Jul 10 00:48:12.210639 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.210554 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed"} err="failed to get container status \"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed\": rpc error: code = NotFound desc = could not find container \"28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed\": container with ID starting with 28828bc8ab95f0c4dacc35f50980d9eacdbef631d48e61a1ef499196f1dfb8ed not found: ID does not exist" Jul 10 00:48:12.286009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.285963 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286064 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286092 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286107 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286127 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286161 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tttgr\" (UniqueName: \"kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr\") pod \"18513eca-f568-4cda-bdef-e4c10a5cfd45\" (UID: \"18513eca-f568-4cda-bdef-e4c10a5cfd45\") " Jul 10 00:48:12.286535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286381 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache" (OuterVolumeSpecName: "model-cache") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:12.286535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286392 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home" (OuterVolumeSpecName: "home") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:12.286535 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286414 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:12.286716 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286549 2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.286716 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286566 2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.286716 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.286581 2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.288546 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.288522 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr" (OuterVolumeSpecName: "kube-api-access-tttgr") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "kube-api-access-tttgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:48:12.288783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.288760 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:48:12.288783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.288760 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm" (OuterVolumeSpecName: "dshm") pod "18513eca-f568-4cda-bdef-e4c10a5cfd45" (UID: "18513eca-f568-4cda-bdef-e4c10a5cfd45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:12.387885 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.387791 2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18513eca-f568-4cda-bdef-e4c10a5cfd45-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.387885 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.387825 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttgr\" (UniqueName: \"kubernetes.io/projected/18513eca-f568-4cda-bdef-e4c10a5cfd45-kube-api-access-tttgr\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.387885 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.387835 2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18513eca-f568-4cda-bdef-e4c10a5cfd45-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:12.522869 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.522819 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:48:12.528836 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:12.528801 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9a8ef25d/e2e-lifecycle-model-9a8ef25d-kserve-75c968496c-q5mdk"] Jul 10 00:48:14.158779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:14.158740 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" path="/var/lib/kubelet/pods/18513eca-f568-4cda-bdef-e4c10a5cfd45/volumes" Jul 10 00:48:36.243030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:36.242938 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:48:53.759746 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:53.759708 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:48:53.760268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:53.759954 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" containerID="cri-o://5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de" gracePeriod=30 Jul 10 00:48:57.639626 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:57.639591 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:48:58.641929 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:58.641890 2548 patch_prober.go:28] interesting pod/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 00:48:58.642319 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:58.641956 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:48:59.025368 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.025342 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:48:59.117302 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117265 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117319 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117381 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117401 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117416 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117454 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24fjd\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117556 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117529 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117585 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117614 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert\") pod \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\" (UID: \"b0ad1fba-e5c3-4629-811b-468f92bfd84c\") " Jul 10 00:48:59.117908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117736 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:59.117908 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117749 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:59.118115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117913 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.118115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117932 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.118115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.117969 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:59.118244 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.118195 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:48:59.118444 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.118422 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:59.120014 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.119989 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:48:59.120087 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.119990 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:48:59.120142 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.120128 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:48:59.120208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.120150 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd" (OuterVolumeSpecName: "kube-api-access-24fjd") pod "b0ad1fba-e5c3-4629-811b-468f92bfd84c" (UID: "b0ad1fba-e5c3-4629-811b-468f92bfd84c"). InnerVolumeSpecName "kube-api-access-24fjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:48:59.218532 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218484 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218532 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218529 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218532 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218541 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fjd\" (UniqueName: \"kubernetes.io/projected/b0ad1fba-e5c3-4629-811b-468f92bfd84c-kube-api-access-24fjd\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218936 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218551 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218936 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218560 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ad1fba-e5c3-4629-811b-468f92bfd84c-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218936 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218569 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.218936 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.218577 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ad1fba-e5c3-4629-811b-468f92bfd84c-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:48:59.438289 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.438254 2548 generic.go:356] "Generic (PLEG): container finished" podID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerID="5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de" exitCode=0 Jul 10 00:48:59.438468 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.438308 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" Jul 10 00:48:59.438468 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.438304 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerDied","Data":"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de"} Jul 10 00:48:59.438468 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.438381 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65" event={"ID":"b0ad1fba-e5c3-4629-811b-468f92bfd84c","Type":"ContainerDied","Data":"650528d071da480cadd7e5ad5a3f20ad967f4b1ac96816da9cae1d91885c3dc8"} Jul 10 00:48:59.438468 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.438414 2548 scope.go:117] "RemoveContainer" containerID="5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de" Jul 10 00:48:59.447611 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.447588 2548 scope.go:117] "RemoveContainer" containerID="02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de" Jul 10 00:48:59.455537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.455517 2548 scope.go:117] "RemoveContainer" containerID="5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de" Jul 10 00:48:59.455801 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:48:59.455782 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de\": container with ID starting with 5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de not found: ID does not exist" containerID="5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de" Jul 10 00:48:59.455868 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.455810 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de"} err="failed to get container status \"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de\": rpc error: code = NotFound desc = could not find container \"5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de\": container with ID starting with 5d8aab30d56bbe7ed5d60ea79c8e011d52d3a09086683dbdba74404db98690de not found: ID does not exist" Jul 10 00:48:59.455868 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.455828 2548 scope.go:117] "RemoveContainer" containerID="02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de" Jul 10 00:48:59.456057 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:48:59.456040 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de\": container with ID starting with 02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de not found: ID does not exist" containerID="02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de" Jul 10 00:48:59.456098 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.456060 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de"} err="failed to get container status \"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de\": rpc error: code = NotFound desc = could not find container \"02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de\": container with ID starting with 02d9e46cabbb1ff29a3a01e42e24f96fb2523502bea68e4c76118f562a3b65de not found: ID does not exist" Jul 10 00:48:59.460041 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.460014 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:48:59.465953 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:48:59.465928 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9a8ef25d-openshift-default-968ddc9f9-m5n65"] Jul 10 00:49:00.159180 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:00.159139 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" path="/var/lib/kubelet/pods/b0ad1fba-e5c3-4629-811b-468f92bfd84c/volumes" Jul 10 00:49:12.246724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:12.246684 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:49:48.288113 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288068 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:49:48.288724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288704 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.288772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288726 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.288772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288742 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.288772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288750 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.288875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288774 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerName="main" Jul 10 00:49:48.288875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288783 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerName="main" Jul 10 00:49:48.288875 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288871 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="18513eca-f568-4cda-bdef-e4c10a5cfd45" containerName="main" Jul 10 00:49:48.288975 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288885 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.288975 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.288896 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ad1fba-e5c3-4629-811b-468f92bfd84c" containerName="istio-proxy" Jul 10 00:49:48.292329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.292300 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.295063 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.295039 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:49:48.295206 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.295044 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b8817a-openshift-default-dockercfg-b6k92\"" Jul 10 00:49:48.304744 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.304709 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:49:48.371669 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371633 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371669 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371667 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371686 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371775 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371821 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371844 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7h9\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.371894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371865 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.372075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371929 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.372075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.371971 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472521 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472569 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472595 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472647 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472672 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472697 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7h9\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.472854 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472835 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472905 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.472934 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.473003 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473264 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.473103 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473264 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.473232 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473374 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.473352 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.473434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.473354 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.475104 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.475075 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.475354 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.475335 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.480790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.480761 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7h9\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.480909 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.480860 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token\") pod \"e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.604853 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.604751 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:48.745180 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.745153 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:49:48.747336 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:49:48.747303 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adddb8d_b382_4325_875f_b1ee810ecd7c.slice/crio-c682e28d0e777ec936b0127812328f34b6d42c9339a2f86c7362bf961f98ffc8 WatchSource:0}: Error finding container c682e28d0e777ec936b0127812328f34b6d42c9339a2f86c7362bf961f98ffc8: Status 404 returned error can't find the container with id c682e28d0e777ec936b0127812328f34b6d42c9339a2f86c7362bf961f98ffc8 Jul 10 00:49:48.749435 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.749405 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:48.749571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.749472 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:48.749571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:48.749518 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:49.697417 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:49.697376 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerStarted","Data":"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668"} Jul 10 00:49:49.697847 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:49.697446 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerStarted","Data":"c682e28d0e777ec936b0127812328f34b6d42c9339a2f86c7362bf961f98ffc8"} Jul 10 00:49:49.717917 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:49.717858 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podStartSLOduration=1.7178378250000002 podStartE2EDuration="1.717837825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:49:49.715596007 +0000 UTC m=+2048.195483559" watchObservedRunningTime="2026-07-10 00:49:49.717837825 +0000 UTC m=+2048.197725373" Jul 10 00:49:50.605444 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:50.605397 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:50.610270 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:50.610240 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:50.703453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:50.703426 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:50.704429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:50.704412 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:49:56.876993 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.876956 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:49:56.880881 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.880855 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.884050 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.884014 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f86ddd-openshift-default-dockercfg-8mb8t\"" Jul 10 00:49:56.892699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.892663 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:49:56.957845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.957797 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.957854 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.957934 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.957980 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.958009 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958053 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.958033 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlntj\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958262 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.958063 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958262 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.958176 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:56.958262 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:56.958212 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059410 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059363 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059410 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059418 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059446 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059483 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059583 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059749 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059799 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.059862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059829 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlntj\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060008 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.059866 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060088 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.060057 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.060089 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.060135 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.060289 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.060711 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.060687 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.062481 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.062432 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.062845 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.062826 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.068410 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.068379 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.068589 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.068440 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlntj\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj\") pod \"e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.194125 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.194027 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:57.336348 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.336321 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:49:57.338703 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:49:57.338667 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae79dc3_7855_453a_9008_637460ab50eb.slice/crio-44fd5a4f87896c7e1b755e463ef3ece31a16edc715c3d5fdacde4a9acf7bd378 WatchSource:0}: Error finding container 44fd5a4f87896c7e1b755e463ef3ece31a16edc715c3d5fdacde4a9acf7bd378: Status 404 returned error can't find the container with id 44fd5a4f87896c7e1b755e463ef3ece31a16edc715c3d5fdacde4a9acf7bd378 Jul 10 00:49:57.341356 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.341323 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:57.341461 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.341394 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:57.341461 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.341423 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:49:57.740354 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.740309 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerStarted","Data":"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36"} Jul 10 00:49:57.740558 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.740370 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerStarted","Data":"44fd5a4f87896c7e1b755e463ef3ece31a16edc715c3d5fdacde4a9acf7bd378"} Jul 10 00:49:57.762776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:57.762720 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podStartSLOduration=1.762701913 podStartE2EDuration="1.762701913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:49:57.760331621 +0000 UTC m=+2056.240219170" watchObservedRunningTime="2026-07-10 00:49:57.762701913 +0000 UTC m=+2056.242589461" Jul 10 00:49:58.194301 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:58.194208 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:58.199215 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:58.199189 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:58.745858 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:58.745822 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:58.746980 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:58.746955 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:49:59.244024 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:49:59.243981 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:50:06.744703 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:06.744656 2548 patch_prober.go:28] interesting pod/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 00:50:06.745212 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:06.744739 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 00:50:07.797038 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.796954 2548 generic.go:356] "Generic (PLEG): container finished" podID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerID="411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668" exitCode=0 Jul 10 00:50:07.797438 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.797032 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerDied","Data":"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668"} Jul 10 00:50:07.797554 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.797518 2548 scope.go:117] "RemoveContainer" containerID="411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668" Jul 10 00:50:07.798637 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.798611 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:07.798701 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.798678 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:07.798741 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:07.798711 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:08.811518 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:08.808007 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerStarted","Data":"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64"} Jul 10 00:50:09.605618 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:09.605445 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:50:09.605808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:09.605680 2548 patch_prober.go:28] interesting pod/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 00:50:09.605808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:09.605727 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 00:50:10.605332 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:10.605287 2548 patch_prober.go:28] interesting pod/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 00:50:10.605778 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:10.605344 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 00:50:11.147721 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:11.147666 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:50:11.605678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:11.605635 2548 patch_prober.go:28] interesting pod/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 00:50:11.606191 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:11.605714 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 00:50:12.605396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:12.605352 2548 patch_prober.go:28] interesting pod/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 00:50:12.605632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:12.605438 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 00:50:13.608800 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:13.608764 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:50:13.609261 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:13.609248 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:50:13.610104 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:13.610083 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:50:14.379541 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.379483 2548 patch_prober.go:28] interesting pod/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:33756->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 10 00:50:14.379683 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.379594 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:33756->10.133.0.55:15021: read: connection reset by peer" Jul 10 00:50:14.840565 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.840532 2548 generic.go:356] "Generic (PLEG): container finished" podID="2ae79dc3-7855-453a-9008-637460ab50eb" containerID="b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36" exitCode=0 Jul 10 00:50:14.841048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.840614 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerDied","Data":"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36"} Jul 10 00:50:14.841097 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.841064 2548 scope.go:117] "RemoveContainer" containerID="b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36" Jul 10 00:50:14.842082 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.842051 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:14.842141 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.842115 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:14.842181 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:14.842147 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:50:15.858071 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:15.858031 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerStarted","Data":"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0"} Jul 10 00:50:16.194609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:16.194518 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:50:16.194808 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:16.194787 2548 patch_prober.go:28] interesting pod/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 00:50:16.194865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:16.194847 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 00:50:17.195204 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:17.195171 2548 patch_prober.go:28] interesting pod/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 00:50:17.195632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:17.195235 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 00:50:18.194394 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:18.194355 2548 patch_prober.go:28] interesting pod/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 00:50:18.194604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:18.194408 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 00:50:19.195027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:19.194984 2548 patch_prober.go:28] interesting pod/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 00:50:19.195577 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:19.195053 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 00:50:20.198366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:20.198335 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:50:20.198858 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:20.198653 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:50:20.199466 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:20.199444 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:50:27.034122 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:27.034081 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:50:38.247756 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:38.247713 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:50:42.196567 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:42.196537 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:50:42.205523 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:50:42.205483 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:51:03.392239 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.392204 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:03.395825 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.395807 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.398608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.398575 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f86ddd\"/\"openshift-service-ca.crt\"" Jul 10 00:51:03.398608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.398604 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f86ddd\"/\"test-model-f86ddd-kserve-self-signed-certs\"" Jul 10 00:51:03.399931 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.399906 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f86ddd\"/\"kube-root-ca.crt\"" Jul 10 00:51:03.400058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.399965 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f86ddd\"/\"default-dockercfg-wrdkk\"" Jul 10 00:51:03.405897 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.405860 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:03.417257 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417221 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.417454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417287 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.417454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417311 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.417454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417361 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.417454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417391 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlfn\" (UniqueName: \"kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.417454 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.417419 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518563 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518801 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518624 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518801 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518644 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518801 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518687 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518801 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518709 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlfn\" (UniqueName: \"kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.518801 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.518739 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.519105 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.519084 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.519164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.519106 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.519164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.519144 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.520890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.520861 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.521271 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.521250 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.527343 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.527316 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlfn\" (UniqueName: \"kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn\") pod \"test-model-f86ddd-kserve-745b8fd685-4wbjc\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.707757 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.707716 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:03.846863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:03.846833 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:03.849397 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:51:03.849362 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod790e5948_24be_46ef_be79_3bd3a4d5db71.slice/crio-3faf9af68c6c12cd5c0d1c40f533611ecefe7dd61ef37c1bb8823312270f4902 WatchSource:0}: Error finding container 3faf9af68c6c12cd5c0d1c40f533611ecefe7dd61ef37c1bb8823312270f4902: Status 404 returned error can't find the container with id 3faf9af68c6c12cd5c0d1c40f533611ecefe7dd61ef37c1bb8823312270f4902 Jul 10 00:51:04.110220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:04.110117 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" event={"ID":"790e5948-24be-46ef-be79-3bd3a4d5db71","Type":"ContainerStarted","Data":"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666"} Jul 10 00:51:04.110220 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:04.110168 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" event={"ID":"790e5948-24be-46ef-be79-3bd3a4d5db71","Type":"ContainerStarted","Data":"3faf9af68c6c12cd5c0d1c40f533611ecefe7dd61ef37c1bb8823312270f4902"} Jul 10 00:51:04.129509 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:04.129443 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" podStartSLOduration=1.129425004 podStartE2EDuration="1.129425004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:51:04.127927298 +0000 UTC m=+2122.607814837" watchObservedRunningTime="2026-07-10 00:51:04.129425004 +0000 UTC m=+2122.609312552" Jul 10 00:51:07.939720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:07.939679 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:51:13.708262 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:13.708216 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:13.708849 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:13.708285 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:13.720890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:13.720858 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:14.179192 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:14.179099 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:21.934884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:21.934843 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:51:24.229990 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:24.229942 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:36.938769 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:36.938682 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:51:39.252713 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.252666 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" podUID="790e5948-24be-46ef-be79-3bd3a4d5db71" containerName="main" containerID="cri-o://00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666" gracePeriod=45 Jul 10 00:51:39.516429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.516357 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:39.634183 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634141 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vlfn\" (UniqueName: \"kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634183 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634190 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634219 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634243 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634272 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634288 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs\") pod \"790e5948-24be-46ef-be79-3bd3a4d5db71\" (UID: \"790e5948-24be-46ef-be79-3bd3a4d5db71\") " Jul 10 00:51:39.634686 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634558 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home" (OuterVolumeSpecName: "home") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:39.634686 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634602 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:39.634686 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.634635 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache" (OuterVolumeSpecName: "model-cache") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:39.636617 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.636583 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn" (OuterVolumeSpecName: "kube-api-access-9vlfn") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "kube-api-access-9vlfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:51:39.636715 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.636607 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm" (OuterVolumeSpecName: "dshm") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:51:39.636715 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.636621 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "790e5948-24be-46ef-be79-3bd3a4d5db71" (UID: "790e5948-24be-46ef-be79-3bd3a4d5db71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:51:39.735512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735448 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlfn\" (UniqueName: \"kubernetes.io/projected/790e5948-24be-46ef-be79-3bd3a4d5db71-kube-api-access-9vlfn\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.735512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735486 2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.735512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735525 2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.735752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735534 2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.735752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735542 2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/790e5948-24be-46ef-be79-3bd3a4d5db71-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:39.735752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:39.735551 2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/790e5948-24be-46ef-be79-3bd3a4d5db71-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:51:40.306035 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.306001 2548 generic.go:356] "Generic (PLEG): container finished" podID="790e5948-24be-46ef-be79-3bd3a4d5db71" containerID="00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666" exitCode=0 Jul 10 00:51:40.306552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.306052 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" Jul 10 00:51:40.306552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.306078 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" event={"ID":"790e5948-24be-46ef-be79-3bd3a4d5db71","Type":"ContainerDied","Data":"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666"} Jul 10 00:51:40.306552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.306125 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc" event={"ID":"790e5948-24be-46ef-be79-3bd3a4d5db71","Type":"ContainerDied","Data":"3faf9af68c6c12cd5c0d1c40f533611ecefe7dd61ef37c1bb8823312270f4902"} Jul 10 00:51:40.306552 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.306142 2548 scope.go:117] "RemoveContainer" containerID="00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666" Jul 10 00:51:40.315630 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.315610 2548 scope.go:117] "RemoveContainer" containerID="00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666" Jul 10 00:51:40.315960 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:51:40.315939 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666\": container with ID starting with 00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666 not found: ID does not exist" containerID="00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666" Jul 10 00:51:40.316030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.315974 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666"} err="failed to get container status \"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666\": rpc error: code = NotFound desc = could not find container \"00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666\": container with ID starting with 00fa8e55124ed6d609ae00239834cdbd15a6c21d81c458e6dc6d4d7ad2229666 not found: ID does not exist" Jul 10 00:51:40.324823 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.324793 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:40.330228 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:40.330197 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f86ddd/test-model-f86ddd-kserve-745b8fd685-4wbjc"] Jul 10 00:51:42.158951 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:51:42.158914 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790e5948-24be-46ef-be79-3bd3a4d5db71" path="/var/lib/kubelet/pods/790e5948-24be-46ef-be79-3bd3a4d5db71/volumes" Jul 10 00:52:06.779437 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:06.779400 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:52:06.779996 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:06.779670 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" containerID="cri-o://34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64" gracePeriod=30 Jul 10 00:52:11.443088 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:11.443044 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:52:12.044152 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.044077 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:52:12.139026 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.138993 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139026 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139029 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139062 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139091 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl7h9\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139115 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139269 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139140 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139482 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139326 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139482 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139370 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:12.139482 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139377 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139649 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139520 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:52:12.139649 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139535 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket\") pod \"1adddb8d-b382-4325-875f-b1ee810ecd7c\" (UID: \"1adddb8d-b382-4325-875f-b1ee810ecd7c\") " Jul 10 00:52:12.139649 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139558 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:12.139836 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.139814 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:12.140044 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.140023 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.140132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.140047 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.140132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.140061 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1adddb8d-b382-4325-875f-b1ee810ecd7c-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.140132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.140083 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.140293 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.140237 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:12.141687 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.141655 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:12.141810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.141746 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:52:12.141862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.141833 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9" (OuterVolumeSpecName: "kube-api-access-nl7h9") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "kube-api-access-nl7h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:12.142028 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.142009 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "1adddb8d-b382-4325-875f-b1ee810ecd7c" (UID: "1adddb8d-b382-4325-875f-b1ee810ecd7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:12.240883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.240848 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.240883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.240880 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7h9\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-kube-api-access-nl7h9\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.240883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.240889 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.241132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.240898 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.241132 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.240907 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1adddb8d-b382-4325-875f-b1ee810ecd7c-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:12.470164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.470127 2548 generic.go:356] "Generic (PLEG): container finished" podID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerID="34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64" exitCode=0 Jul 10 00:52:12.470604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.470175 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" Jul 10 00:52:12.470604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.470198 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerDied","Data":"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64"} Jul 10 00:52:12.470604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.470248 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r" event={"ID":"1adddb8d-b382-4325-875f-b1ee810ecd7c","Type":"ContainerDied","Data":"c682e28d0e777ec936b0127812328f34b6d42c9339a2f86c7362bf961f98ffc8"} Jul 10 00:52:12.470604 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.470273 2548 scope.go:117] "RemoveContainer" containerID="34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64" Jul 10 00:52:12.479776 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.479757 2548 scope.go:117] "RemoveContainer" containerID="411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668" Jul 10 00:52:12.487761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.487735 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:52:12.488640 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.488614 2548 scope.go:117] "RemoveContainer" containerID="34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64" Jul 10 00:52:12.489158 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:12.489018 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64\": container with ID starting with 34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64 not found: ID does not exist" containerID="34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64" Jul 10 00:52:12.489158 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.489053 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64"} err="failed to get container status \"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64\": rpc error: code = NotFound desc = could not find container \"34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64\": container with ID starting with 34f92709d76adaa2cd5651e69994a6a89aae9039c614a47a1f07c48e7f8cee64 not found: ID does not exist" Jul 10 00:52:12.489158 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.489079 2548 scope.go:117] "RemoveContainer" containerID="411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668" Jul 10 00:52:12.489473 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:12.489436 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668\": container with ID starting with 411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668 not found: ID does not exist" containerID="411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668" Jul 10 00:52:12.489802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.489471 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668"} err="failed to get container status \"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668\": rpc error: code = NotFound desc = could not find container \"411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668\": container with ID starting with 411bc0aa0cebc17b3efa6a37feb3414958134cd63e029a600aeef127ec7bd668 not found: ID does not exist" Jul 10 00:52:12.491458 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:12.491438 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b8817a-openshift-default-66ff54ffdc-hwh4r"] Jul 10 00:52:13.555843 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:13.555811 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:52:13.556361 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:13.556031 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" containerID="cri-o://afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0" gracePeriod=30 Jul 10 00:52:14.158150 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.158115 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" path="/var/lib/kubelet/pods/1adddb8d-b382-4325-875f-b1ee810ecd7c/volumes" Jul 10 00:52:14.434571 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.434385 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:14.435019 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.434987 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790e5948-24be-46ef-be79-3bd3a4d5db71" containerName="main" Jul 10 00:52:14.435019 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435016 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="790e5948-24be-46ef-be79-3bd3a4d5db71" containerName="main" Jul 10 00:52:14.435233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435054 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:14.435233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435063 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:14.435233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435170 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:14.435233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435182 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:14.435233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.435189 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="790e5948-24be-46ef-be79-3bd3a4d5db71" containerName="main" Jul 10 00:52:14.439960 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.439935 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.442739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.442717 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f7d727-openshift-default-dockercfg-979wd\"" Jul 10 00:52:14.449161 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.449103 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:14.562396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562356 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562398 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562518 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562551 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562601 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562640 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562661 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562680 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxfm\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.562867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.562726 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.663754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663712 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.663754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663756 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663806 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663822 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663855 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663877 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663901 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.663928 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxfm\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664298 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664267 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664359 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664295 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664460 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664438 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664475 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664649 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.664755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.664731 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.666385 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.666361 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.666760 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.666740 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.671664 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.671641 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.671835 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.671812 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxfm\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm\") pod \"e2e-api-a-f7d727-openshift-default-674b59c749-88mth\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.751781 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.751735 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:14.906149 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.906038 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:14.908403 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:52:14.908380 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2787e4bd_0893_42fa_ab83_72c8c63f1154.slice/crio-08caa34be6eb670ccc3d8741b93fe12a92ce01d91aae770443ff3703fe0d9d6c WatchSource:0}: Error finding container 08caa34be6eb670ccc3d8741b93fe12a92ce01d91aae770443ff3703fe0d9d6c: Status 404 returned error can't find the container with id 08caa34be6eb670ccc3d8741b93fe12a92ce01d91aae770443ff3703fe0d9d6c Jul 10 00:52:14.910675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.910642 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:14.910784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.910725 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:14.910784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:14.910768 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:15.488245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:15.488201 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerStarted","Data":"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a"} Jul 10 00:52:15.488423 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:15.488265 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerStarted","Data":"08caa34be6eb670ccc3d8741b93fe12a92ce01d91aae770443ff3703fe0d9d6c"} Jul 10 00:52:15.507463 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:15.507410 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podStartSLOduration=1.507392412 podStartE2EDuration="1.507392412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:52:15.506519123 +0000 UTC m=+2193.986406673" watchObservedRunningTime="2026-07-10 00:52:15.507392412 +0000 UTC m=+2193.987279964" Jul 10 00:52:15.752486 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:15.752379 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:15.757672 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:15.757640 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:16.493252 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:16.493215 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:16.494170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:16.494151 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:18.818439 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.818360 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:52:18.903812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903774 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.903812 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903820 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlntj\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903861 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903897 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903920 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903936 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903953 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.903984 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904012 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo\") pod \"2ae79dc3-7855-453a-9008-637460ab50eb\" (UID: \"2ae79dc3-7855-453a-9008-637460ab50eb\") " Jul 10 00:52:18.904443 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904143 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:18.904443 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904387 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:18.904443 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904427 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:18.904631 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904549 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:52:18.904706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.904688 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:18.905062 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.905025 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:18.906431 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.906400 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj" (OuterVolumeSpecName: "kube-api-access-hlntj") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "kube-api-access-hlntj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:18.906546 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.906443 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:18.906757 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.906735 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:52:18.906867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:18.906842 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ae79dc3-7855-453a-9008-637460ab50eb" (UID: "2ae79dc3-7855-453a-9008-637460ab50eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:19.005405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005369 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005401 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005410 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005405 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005418 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005427 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ae79dc3-7855-453a-9008-637460ab50eb-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005434 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ae79dc3-7855-453a-9008-637460ab50eb-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005442 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ae79dc3-7855-453a-9008-637460ab50eb-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.005788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.005450 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlntj\" (UniqueName: \"kubernetes.io/projected/2ae79dc3-7855-453a-9008-637460ab50eb-kube-api-access-hlntj\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:19.512439 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.512400 2548 generic.go:356] "Generic (PLEG): container finished" podID="2ae79dc3-7855-453a-9008-637460ab50eb" containerID="afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0" exitCode=0 Jul 10 00:52:19.512439 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.512448 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" Jul 10 00:52:19.512739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.512471 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerDied","Data":"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0"} Jul 10 00:52:19.512739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.512560 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx" event={"ID":"2ae79dc3-7855-453a-9008-637460ab50eb","Type":"ContainerDied","Data":"44fd5a4f87896c7e1b755e463ef3ece31a16edc715c3d5fdacde4a9acf7bd378"} Jul 10 00:52:19.512739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.512594 2548 scope.go:117] "RemoveContainer" containerID="afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0" Jul 10 00:52:19.522758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.522732 2548 scope.go:117] "RemoveContainer" containerID="b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36" Jul 10 00:52:19.532876 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.532822 2548 scope.go:117] "RemoveContainer" containerID="afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0" Jul 10 00:52:19.533281 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:19.533243 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0\": container with ID starting with afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0 not found: ID does not exist" containerID="afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0" Jul 10 00:52:19.533415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.533292 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0"} err="failed to get container status \"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0\": rpc error: code = NotFound desc = could not find container \"afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0\": container with ID starting with afb7c1b2de38b13fd557f7e203c21d744e522edf8c0f38d3ba48bbe00947dac0 not found: ID does not exist" Jul 10 00:52:19.533415 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.533317 2548 scope.go:117] "RemoveContainer" containerID="b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36" Jul 10 00:52:19.533667 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:19.533647 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36\": container with ID starting with b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36 not found: ID does not exist" containerID="b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36" Jul 10 00:52:19.533728 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.533674 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36"} err="failed to get container status \"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36\": rpc error: code = NotFound desc = could not find container \"b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36\": container with ID starting with b0a9990039306084c6f2d375bd7cd0e726262e7de4b982848b7d35f24f54bd36 not found: ID does not exist" Jul 10 00:52:19.534790 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.534769 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:52:19.538401 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:19.538373 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f86ddd-openshift-default-74fb58b7ff-mfbhx"] Jul 10 00:52:20.158483 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:20.158451 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" path="/var/lib/kubelet/pods/2ae79dc3-7855-453a-9008-637460ab50eb/volumes" Jul 10 00:52:22.670100 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670066 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:22.670587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670551 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:22.670587 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670570 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="1adddb8d-b382-4325-875f-b1ee810ecd7c" containerName="istio-proxy" Jul 10 00:52:22.670723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670583 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.670723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670631 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.670723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670645 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.670723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670654 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.670913 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670819 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.670913 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.670833 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae79dc3-7855-453a-9008-637460ab50eb" containerName="istio-proxy" Jul 10 00:52:22.675672 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.675649 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.678318 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.678296 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0af175-openshift-default-dockercfg-t7mc9\"" Jul 10 00:52:22.685861 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.685833 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:22.740075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740037 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740075 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740072 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740100 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740197 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8trp\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740245 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740424 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740331 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740424 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740359 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740424 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740397 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.740573 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.740425 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841265 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841214 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841265 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841254 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841290 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841418 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841536 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841568 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841606 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841645 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8trp\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841691 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.841803 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.841697 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.842057 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.842027 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.842233 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.842211 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.842307 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.842293 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.842664 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.842640 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.844035 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.844004 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.844247 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.844230 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.854699 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.854666 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.854865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.854841 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8trp\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp\") pod \"e2e-api-b-0af175-openshift-default-8454779649-j6c84\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:22.986890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:22.986847 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:23.136624 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.136579 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:23.140365 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:52:23.140322 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dfbb8b_a228_451f_97dc_7cad3d09fb0b.slice/crio-14875eca199677c327b3b8cb1c63843f1bb688bb36021d54e5d3785faa0a824c WatchSource:0}: Error finding container 14875eca199677c327b3b8cb1c63843f1bb688bb36021d54e5d3785faa0a824c: Status 404 returned error can't find the container with id 14875eca199677c327b3b8cb1c63843f1bb688bb36021d54e5d3785faa0a824c Jul 10 00:52:23.142987 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.142945 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:23.143090 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.143032 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:23.143090 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.143076 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:23.535927 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.535878 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerStarted","Data":"eca28c5bc73c0048c45a79db97df23b6f49d474767a3e2ec45721e0cbb880ead"} Jul 10 00:52:23.536119 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.535953 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerStarted","Data":"14875eca199677c327b3b8cb1c63843f1bb688bb36021d54e5d3785faa0a824c"} Jul 10 00:52:23.556802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.556739 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podStartSLOduration=1.556718387 podStartE2EDuration="1.556718387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:52:23.555548239 +0000 UTC m=+2202.035435785" watchObservedRunningTime="2026-07-10 00:52:23.556718387 +0000 UTC m=+2202.036605936" Jul 10 00:52:23.987324 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:23.987285 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:24.993021 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:24.992987 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:25.545752 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:25.545720 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:25.546731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:25.546707 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:25.557479 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:25.557447 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:52:32.534722 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:32.534675 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 00:52:32.535204 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:32.534751 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 00:52:34.596346 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.596303 2548 generic.go:356] "Generic (PLEG): container finished" podID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerID="4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a" exitCode=0 Jul 10 00:52:34.596809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.596377 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerDied","Data":"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a"} Jul 10 00:52:34.596857 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.596825 2548 scope.go:117] "RemoveContainer" containerID="4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a" Jul 10 00:52:34.597920 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.597893 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:34.597980 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.597968 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:34.598018 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:34.598001 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:35.604148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:35.604108 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerStarted","Data":"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d"} Jul 10 00:52:35.752891 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:35.752843 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:35.753106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:35.752987 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 00:52:35.753106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:35.753034 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 00:52:36.752253 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:36.752210 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 00:52:36.752914 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:36.752878 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 00:52:37.753200 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:37.753158 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 00:52:37.753899 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:37.753865 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 00:52:38.450434 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:38.450320 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:52:38.752598 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:38.752561 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 00:52:38.752906 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:38.752876 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 00:52:39.753192 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:39.752740 2548 patch_prober.go:28] interesting pod/e2e-api-a-f7d727-openshift-default-674b59c749-88mth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 00:52:39.753192 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:39.752806 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 00:52:40.756395 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:40.756361 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:40.756851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:40.756790 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:40.757820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:40.757801 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:41.222278 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.222178 2548 patch_prober.go:28] interesting pod/e2e-api-b-0af175-openshift-default-8454779649-j6c84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:53246->10.133.0.58:15021: read: connection reset by peer" start-of-body= Jul 10 00:52:41.222278 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.222241 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:53246->10.133.0.58:15021: read: connection reset by peer" Jul 10 00:52:41.636010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.635925 2548 generic.go:356] "Generic (PLEG): container finished" podID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerID="eca28c5bc73c0048c45a79db97df23b6f49d474767a3e2ec45721e0cbb880ead" exitCode=0 Jul 10 00:52:41.636164 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.636006 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerDied","Data":"eca28c5bc73c0048c45a79db97df23b6f49d474767a3e2ec45721e0cbb880ead"} Jul 10 00:52:41.636526 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.636513 2548 scope.go:117] "RemoveContainer" containerID="eca28c5bc73c0048c45a79db97df23b6f49d474767a3e2ec45721e0cbb880ead" Jul 10 00:52:41.637531 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.637482 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:41.637614 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.637596 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:41.637677 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:41.637662 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:42.656389 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:42.656285 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerStarted","Data":"ee7a5eac7f2db24dd0f45aef2905b1ecc147e8e24611425bf878e5614ad1527b"} Jul 10 00:52:42.987895 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:42.987850 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:42.988145 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:42.988121 2548 patch_prober.go:28] interesting pod/e2e-api-b-0af175-openshift-default-8454779649-j6c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:52:42.988203 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:42.988181 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 00:52:43.987870 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:43.987831 2548 patch_prober.go:28] interesting pod/e2e-api-b-0af175-openshift-default-8454779649-j6c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:52:43.988288 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:43.987888 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 00:52:44.813649 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:44.813613 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:44.813851 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:44.813822 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" containerID="cri-o://f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d" gracePeriod=30 Jul 10 00:52:44.987842 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:44.987809 2548 patch_prober.go:28] interesting pod/e2e-api-b-0af175-openshift-default-8454779649-j6c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:52:44.988029 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:44.987863 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 00:52:45.987431 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:45.987394 2548 patch_prober.go:28] interesting pod/e2e-api-b-0af175-openshift-default-8454779649-j6c84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:52:45.987655 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:45.987447 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 10 00:52:47.004325 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:47.004294 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:47.004768 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:47.004714 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:47.005474 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:47.005454 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:49.050584 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:49.050546 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:52:50.080775 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.080708 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:50.110734 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110700 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110734 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110735 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110757 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110774 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110810 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110834 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110856 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcxfm\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110892 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.110978 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.110965 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert\") pod \"2787e4bd-0893-42fa-ab83-72c8c63f1154\" (UID: \"2787e4bd-0893-42fa-ab83-72c8c63f1154\") " Jul 10 00:52:50.111326 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111038 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:50.111326 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111114 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:50.111326 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111232 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:50.111477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111351 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.111477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111372 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.111477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111424 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.111647 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.111559 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:52:50.112336 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.112305 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data" (OuterVolumeSpecName: "istio-data") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:50.113457 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.113402 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm" (OuterVolumeSpecName: "kube-api-access-vcxfm") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "kube-api-access-vcxfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:50.113632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.113464 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token" (OuterVolumeSpecName: "istio-token") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:50.113632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.113526 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:52:50.113632 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.113529 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2787e4bd-0893-42fa-ab83-72c8c63f1154" (UID: "2787e4bd-0893-42fa-ab83-72c8c63f1154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:50.212478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212437 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.212478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212471 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.212478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212481 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.212478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212517 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxfm\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-kube-api-access-vcxfm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.212821 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212529 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2787e4bd-0893-42fa-ab83-72c8c63f1154-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.212821 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.212540 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2787e4bd-0893-42fa-ab83-72c8c63f1154-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:50.705964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.705921 2548 generic.go:356] "Generic (PLEG): container finished" podID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerID="f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d" exitCode=0 Jul 10 00:52:50.706176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.705972 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" Jul 10 00:52:50.706176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.706017 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerDied","Data":"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d"} Jul 10 00:52:50.706176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.706082 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth" event={"ID":"2787e4bd-0893-42fa-ab83-72c8c63f1154","Type":"ContainerDied","Data":"08caa34be6eb670ccc3d8741b93fe12a92ce01d91aae770443ff3703fe0d9d6c"} Jul 10 00:52:50.706176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.706117 2548 scope.go:117] "RemoveContainer" containerID="f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d" Jul 10 00:52:50.715550 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.715529 2548 scope.go:117] "RemoveContainer" containerID="4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a" Jul 10 00:52:50.723759 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.723728 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:50.725158 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.725128 2548 scope.go:117] "RemoveContainer" containerID="f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d" Jul 10 00:52:50.725883 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:50.725852 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d\": container with ID starting with f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d not found: ID does not exist" containerID="f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d" Jul 10 00:52:50.726025 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.725895 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d"} err="failed to get container status \"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d\": rpc error: code = NotFound desc = could not find container \"f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d\": container with ID starting with f732c8b4df8d96c0344d6b0b68c8a5d5e58b64b44f85ff52f600056b934b1c3d not found: ID does not exist" Jul 10 00:52:50.726025 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.725922 2548 scope.go:117] "RemoveContainer" containerID="4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a" Jul 10 00:52:50.726300 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:52:50.726280 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a\": container with ID starting with 4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a not found: ID does not exist" containerID="4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a" Jul 10 00:52:50.726418 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.726308 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a"} err="failed to get container status \"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a\": rpc error: code = NotFound desc = could not find container \"4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a\": container with ID starting with 4698edca3a3eba7e6367363f2ec49693b7d903110b4c528e5a77fc63b526c49a not found: ID does not exist" Jul 10 00:52:50.728381 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:50.728349 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f7d727-openshift-default-674b59c749-88mth"] Jul 10 00:52:51.503995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:51.503962 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:51.504428 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:51.504167 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" containerID="cri-o://ee7a5eac7f2db24dd0f45aef2905b1ecc147e8e24611425bf878e5614ad1527b" gracePeriod=30 Jul 10 00:52:52.158320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.158276 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" path="/var/lib/kubelet/pods/2787e4bd-0893-42fa-ab83-72c8c63f1154/volumes" Jul 10 00:52:52.413824 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.413735 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:52:52.414421 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414396 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.414421 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414420 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.414599 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414433 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.414599 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414440 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.414599 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414530 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.414599 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.414538 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="2787e4bd-0893-42fa-ab83-72c8c63f1154" containerName="istio-proxy" Jul 10 00:52:52.417252 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.417232 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.419855 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.419835 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d62432-openshift-default-dockercfg-gw699\"" Jul 10 00:52:52.428014 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.427987 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:52:52.434158 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434130 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434165 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434191 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434218 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434320 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434283 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434394 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434440 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5mx\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434538 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.434596 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.434565 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.535720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535676 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.535720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535728 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5mx\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535852 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535888 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535929 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.535965 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536006 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536054 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536268 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536076 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536391 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536419 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536466 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536623 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536600 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.536820 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.536800 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.538894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.538864 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.539111 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.539093 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.544102 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.544071 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5mx\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.544224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.544169 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token\") pod \"e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.728217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.728171 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:52.873456 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.873427 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:52:52.875426 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:52:52.875389 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab236d6_fc03_48f5_9e66_da6f0b807060.slice/crio-ad819a0809b46f1da6ee4a396e7fa65892731803bb2646099fb4bbef6cc804de WatchSource:0}: Error finding container ad819a0809b46f1da6ee4a396e7fa65892731803bb2646099fb4bbef6cc804de: Status 404 returned error can't find the container with id ad819a0809b46f1da6ee4a396e7fa65892731803bb2646099fb4bbef6cc804de Jul 10 00:52:52.877711 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.877674 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:52.877788 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.877772 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:52.877842 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:52.877825 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:52:53.727143 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:53.727105 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerStarted","Data":"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e"} Jul 10 00:52:53.727546 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:53.727163 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerStarted","Data":"ad819a0809b46f1da6ee4a396e7fa65892731803bb2646099fb4bbef6cc804de"} Jul 10 00:52:53.728389 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:53.728368 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:53.733675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:53.733650 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:53.748600 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:53.748548 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podStartSLOduration=1.74853152 podStartE2EDuration="1.74853152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:52:53.746344312 +0000 UTC m=+2232.226231850" watchObservedRunningTime="2026-07-10 00:52:53.74853152 +0000 UTC m=+2232.228419068" Jul 10 00:52:54.731898 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:54.731863 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:54.733002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:54.732974 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:52:56.745540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.745477 2548 generic.go:356] "Generic (PLEG): container finished" podID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerID="ee7a5eac7f2db24dd0f45aef2905b1ecc147e8e24611425bf878e5614ad1527b" exitCode=0 Jul 10 00:52:56.746022 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.745556 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerDied","Data":"ee7a5eac7f2db24dd0f45aef2905b1ecc147e8e24611425bf878e5614ad1527b"} Jul 10 00:52:56.746022 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.745643 2548 scope.go:117] "RemoveContainer" containerID="eca28c5bc73c0048c45a79db97df23b6f49d474767a3e2ec45721e0cbb880ead" Jul 10 00:52:56.782839 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.782768 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:56.881036 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.880996 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881036 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881038 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881074 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881105 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881122 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881157 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881180 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8trp\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881212 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881254 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket\") pod \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\" (UID: \"70dfbb8b-a228-451f-97dc-7cad3d09fb0b\") " Jul 10 00:52:56.881671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881333 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:56.881671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881484 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:52:56.881671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881604 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:56.881835 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881687 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.881835 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.881799 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:56.882256 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.882231 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:56.883601 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.883569 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:52:56.883720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.883615 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:52:56.883720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.883673 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:56.884017 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.883988 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp" (OuterVolumeSpecName: "kube-api-access-q8trp") pod "70dfbb8b-a228-451f-97dc-7cad3d09fb0b" (UID: "70dfbb8b-a228-451f-97dc-7cad3d09fb0b"). InnerVolumeSpecName "kube-api-access-q8trp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:52:56.982471 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982424 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982471 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982463 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982471 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982473 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982471 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982484 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8trp\" (UniqueName: \"kubernetes.io/projected/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-kube-api-access-q8trp\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982527 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982535 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982543 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:56.982783 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:56.982551 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70dfbb8b-a228-451f-97dc-7cad3d09fb0b-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:52:57.754208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:57.754177 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" Jul 10 00:52:57.754663 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:57.754221 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84" event={"ID":"70dfbb8b-a228-451f-97dc-7cad3d09fb0b","Type":"ContainerDied","Data":"14875eca199677c327b3b8cb1c63843f1bb688bb36021d54e5d3785faa0a824c"} Jul 10 00:52:57.754663 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:57.754280 2548 scope.go:117] "RemoveContainer" containerID="ee7a5eac7f2db24dd0f45aef2905b1ecc147e8e24611425bf878e5614ad1527b" Jul 10 00:52:57.779894 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:57.779863 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:57.783464 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:57.783438 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0af175-openshift-default-8454779649-j6c84"] Jul 10 00:52:58.158295 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:52:58.158211 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" path="/var/lib/kubelet/pods/70dfbb8b-a228-451f-97dc-7cad3d09fb0b/volumes" Jul 10 00:53:00.821625 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.821590 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:53:00.822229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.822203 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:00.822229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.822229 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:00.822391 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.822309 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:00.822391 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.822318 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:00.825176 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.825152 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.827925 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.827895 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f84011-openshift-default-dockercfg-tvqzz\"" Jul 10 00:53:00.835426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.835392 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:53:00.914616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914576 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gt9\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914663 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914699 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914727 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914773 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914799 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.914838 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914817 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.915127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.914965 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:00.915127 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:00.915019 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015566 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015518 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gt9\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015580 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015609 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015640 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015689 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015711 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.015743 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015734 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015808 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.015849 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016192 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.016143 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016259 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.016235 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016259 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.016251 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016371 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.016281 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.016429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.016403 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.018039 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.018015 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.018364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.018341 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.023763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.023736 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gt9\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.023763 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.023746 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token\") pod \"e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.139113 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.139018 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:01.288447 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.288418 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:53:01.290660 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:53:01.290630 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4510c2_873d_4fa2_ba71_d9a7bfda555c.slice/crio-f8ecae8fbb807d789fe564445297393572426e3fd02e2c2aa2170e7ec0424206 WatchSource:0}: Error finding container f8ecae8fbb807d789fe564445297393572426e3fd02e2c2aa2170e7ec0424206: Status 404 returned error can't find the container with id f8ecae8fbb807d789fe564445297393572426e3fd02e2c2aa2170e7ec0424206 Jul 10 00:53:01.292991 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.292956 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:01.293100 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.293041 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:01.293161 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.293105 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:01.778671 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.778635 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerStarted","Data":"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661"} Jul 10 00:53:01.778867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.778687 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerStarted","Data":"f8ecae8fbb807d789fe564445297393572426e3fd02e2c2aa2170e7ec0424206"} Jul 10 00:53:01.801608 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:01.801543 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podStartSLOduration=1.8015166360000001 podStartE2EDuration="1.801516636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:53:01.797773271 +0000 UTC m=+2240.277660832" watchObservedRunningTime="2026-07-10 00:53:01.801516636 +0000 UTC m=+2240.281404180" Jul 10 00:53:02.139440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:02.139342 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:02.144809 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:02.144780 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:02.785000 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:02.784965 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:02.786273 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:02.786253 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:04.141071 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:04.140983 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:53:10.773723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:10.773675 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 00:53:10.774205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:10.773787 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 00:53:12.841092 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.841040 2548 generic.go:356] "Generic (PLEG): container finished" podID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerID="c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e" exitCode=0 Jul 10 00:53:12.841684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.841131 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerDied","Data":"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e"} Jul 10 00:53:12.841684 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.841670 2548 scope.go:117] "RemoveContainer" containerID="c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e" Jul 10 00:53:12.843416 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.843376 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:12.843657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.843634 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:12.843804 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:12.843785 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:13.851988 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:13.851939 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerStarted","Data":"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e"} Jul 10 00:53:14.729245 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:14.729206 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:53:14.729472 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:14.729444 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 00:53:14.729704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:14.729678 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 00:53:15.441858 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:15.441809 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:53:15.729148 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:15.729101 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 00:53:15.729336 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:15.729192 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 00:53:16.728772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:16.728736 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 00:53:16.729209 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:16.728796 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 00:53:17.728724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:17.728684 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 00:53:17.728919 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:17.728746 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 00:53:18.733906 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:18.733872 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:53:18.734360 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:18.734268 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:53:18.735337 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:18.735318 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:53:18.827720 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:18.827687 2548 patch_prober.go:28] interesting pod/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 00:53:18.827915 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:18.827755 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 00:53:19.889087 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.889047 2548 generic.go:356] "Generic (PLEG): container finished" podID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerID="e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661" exitCode=0 Jul 10 00:53:19.889555 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.889121 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerDied","Data":"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661"} Jul 10 00:53:19.889631 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.889614 2548 scope.go:117] "RemoveContainer" containerID="e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661" Jul 10 00:53:19.890767 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.890738 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:19.890865 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.890825 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:19.890918 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:19.890877 2548 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:53:20.896641 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:20.896595 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerStarted","Data":"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d"} Jul 10 00:53:21.139329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:21.139289 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:21.140039 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:21.140001 2548 patch_prober.go:28] interesting pod/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 00:53:21.140186 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:21.140067 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 00:53:22.139953 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:22.139910 2548 patch_prober.go:28] interesting pod/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 00:53:22.140595 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:22.140561 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 00:53:23.140009 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:23.139960 2548 patch_prober.go:28] interesting pod/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 00:53:23.140530 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:23.140051 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 00:53:24.140350 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:24.140307 2548 patch_prober.go:28] interesting pod/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 00:53:24.141417 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:24.140980 2548 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 00:53:25.143785 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:25.143750 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:25.144223 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:25.144031 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:25.144758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:25.144737 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:53:27.096051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.096006 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:53:27.097002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.096977 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:27.097002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.097004 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="70dfbb8b-a228-451f-97dc-7cad3d09fb0b" containerName="istio-proxy" Jul 10 00:53:27.099408 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.099385 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.102224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.102192 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"kube-root-ca.crt\"" Jul 10 00:53:27.103344 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.103309 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"default-dockercfg-tz7ms\"" Jul 10 00:53:27.103476 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.103381 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"auth-test-model-d62432-kserve-self-signed-certs\"" Jul 10 00:53:27.103476 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.103397 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"openshift-service-ca.crt\"" Jul 10 00:53:27.107973 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.107948 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:53:27.208167 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208118 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.208362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208184 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.208362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208247 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.208362 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208337 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zd9\" (UniqueName: \"kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.208478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208379 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.208478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.208458 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.309597 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.309532 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.309807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.309620 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zd9\" (UniqueName: \"kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.309807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.309652 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.309807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.309690 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.310211 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.309744 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.310307 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.310286 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.310377 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.310340 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.317766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.310450 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.317766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.310723 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.317766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.313002 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.317766 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.313278 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.320807 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.320775 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zd9\" (UniqueName: \"kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9\") pod \"auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.409995 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.409879 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:27.550991 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.550950 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:53:27.554378 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:53:27.554345 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09545702_df66_4179_8b9b_4d57e95109b7.slice/crio-cb1b11efcb4a2669c7cd11e1ae4a44e226f5914c09754597f0f7d4eb44601462 WatchSource:0}: Error finding container cb1b11efcb4a2669c7cd11e1ae4a44e226f5914c09754597f0f7d4eb44601462: Status 404 returned error can't find the container with id cb1b11efcb4a2669c7cd11e1ae4a44e226f5914c09754597f0f7d4eb44601462 Jul 10 00:53:27.934516 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.934458 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" event={"ID":"09545702-df66-4179-8b9b-4d57e95109b7","Type":"ContainerStarted","Data":"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983"} Jul 10 00:53:27.934739 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.934528 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" event={"ID":"09545702-df66-4179-8b9b-4d57e95109b7","Type":"ContainerStarted","Data":"cb1b11efcb4a2669c7cd11e1ae4a44e226f5914c09754597f0f7d4eb44601462"} Jul 10 00:53:27.951945 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:27.951892 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" podStartSLOduration=0.951876189 podStartE2EDuration="951.876189ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:53:27.951298448 +0000 UTC m=+2266.431186009" watchObservedRunningTime="2026-07-10 00:53:27.951876189 +0000 UTC m=+2266.431763737" Jul 10 00:53:30.642567 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:30.642526 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:53:37.410208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:37.410152 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:37.410208 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:37.410227 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:37.422770 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:37.422742 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:37.998031 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:37.997994 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:53:51.842960 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:53:51.842922 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:54:07.049758 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:54:07.049718 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:54:25.845358 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:54:25.845322 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:54:40.088643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:54:40.088547 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:54:55.243170 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:54:55.243126 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:55:12.739923 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:12.739884 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:55:27.643643 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:27.643604 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:55:39.191815 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:39.191777 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:55:42.234818 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:42.234775 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:55:42.245512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:42.245465 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:55:43.243924 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:43.243878 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:55:43.983004 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:43.982969 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:55:43.985373 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:43.985356 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:43.987974 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:43.987949 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"sub-test-model-d62432-kserve-self-signed-certs\"" Jul 10 00:55:43.998512 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:43.998467 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:55:44.153944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.153901 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.154157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.153962 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrbjt\" (UniqueName: \"kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.154157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.154009 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.154157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.154037 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.154157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.154064 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.154157 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.154101 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255109 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255011 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255109 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255061 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255125 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255158 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrbjt\" (UniqueName: \"kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255206 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255237 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255574 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255605 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.255675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.255641 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.257402 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.257372 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.257782 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.257764 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.263590 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.263565 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrbjt\" (UniqueName: \"kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt\") pod \"sub-test-model-d62432-kserve-878fddb8f-tcfrd\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.295797 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.295751 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:44.430200 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.430175 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:55:44.432320 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:55:44.432284 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908d2f62_c65f_4bfa_a8b0_c2b03134ef1d.slice/crio-f9e2649996b636a18a3f7f7f997eeb52089baee336d4cd924a640770245074a6 WatchSource:0}: Error finding container f9e2649996b636a18a3f7f7f997eeb52089baee336d4cd924a640770245074a6: Status 404 returned error can't find the container with id f9e2649996b636a18a3f7f7f997eeb52089baee336d4cd924a640770245074a6 Jul 10 00:55:44.634563 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.634451 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" event={"ID":"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d","Type":"ContainerStarted","Data":"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384"} Jul 10 00:55:44.634563 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.634532 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" event={"ID":"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d","Type":"ContainerStarted","Data":"f9e2649996b636a18a3f7f7f997eeb52089baee336d4cd924a640770245074a6"} Jul 10 00:55:44.650704 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:44.650641 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" podStartSLOduration=1.650622579 podStartE2EDuration="1.650622579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:55:44.649525423 +0000 UTC m=+2403.129412974" watchObservedRunningTime="2026-07-10 00:55:44.650622579 +0000 UTC m=+2403.130510126" Jul 10 00:55:54.212106 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.212042 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" podUID="09545702-df66-4179-8b9b-4d57e95109b7" containerName="main" containerID="cri-o://137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983" gracePeriod=45 Jul 10 00:55:54.296077 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.296039 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:54.296238 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.296091 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:54.309537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.309485 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:54.478531 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.478422 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:55:54.559762 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559720 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.559971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559809 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.559971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559841 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.559971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559873 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8zd9\" (UniqueName: \"kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.559971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559906 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.559971 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.559953 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir\") pod \"09545702-df66-4179-8b9b-4d57e95109b7\" (UID: \"09545702-df66-4179-8b9b-4d57e95109b7\") " Jul 10 00:55:54.560216 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560051 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home" (OuterVolumeSpecName: "home") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:54.560216 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560147 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:54.560325 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560298 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:54.560423 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560406 2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.560487 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560424 2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.560487 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.560433 2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.562182 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.562154 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm" (OuterVolumeSpecName: "dshm") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:55:54.562182 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.562158 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:55:54.562346 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.562219 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9" (OuterVolumeSpecName: "kube-api-access-r8zd9") pod "09545702-df66-4179-8b9b-4d57e95109b7" (UID: "09545702-df66-4179-8b9b-4d57e95109b7"). InnerVolumeSpecName "kube-api-access-r8zd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:55:54.662030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.661984 2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09545702-df66-4179-8b9b-4d57e95109b7-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.662030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.662020 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8zd9\" (UniqueName: \"kubernetes.io/projected/09545702-df66-4179-8b9b-4d57e95109b7-kube-api-access-r8zd9\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.662030 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.662035 2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09545702-df66-4179-8b9b-4d57e95109b7-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:55:54.688644 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.688596 2548 generic.go:356] "Generic (PLEG): container finished" podID="09545702-df66-4179-8b9b-4d57e95109b7" containerID="137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983" exitCode=0 Jul 10 00:55:54.688644 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.688647 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" Jul 10 00:55:54.688905 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.688688 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" event={"ID":"09545702-df66-4179-8b9b-4d57e95109b7","Type":"ContainerDied","Data":"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983"} Jul 10 00:55:54.688905 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.688730 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6" event={"ID":"09545702-df66-4179-8b9b-4d57e95109b7","Type":"ContainerDied","Data":"cb1b11efcb4a2669c7cd11e1ae4a44e226f5914c09754597f0f7d4eb44601462"} Jul 10 00:55:54.688905 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.688752 2548 scope.go:117] "RemoveContainer" containerID="137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983" Jul 10 00:55:54.698256 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.698234 2548 scope.go:117] "RemoveContainer" containerID="137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983" Jul 10 00:55:54.698633 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:55:54.698610 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983\": container with ID starting with 137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983 not found: ID does not exist" containerID="137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983" Jul 10 00:55:54.698748 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.698645 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983"} err="failed to get container status \"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983\": rpc error: code = NotFound desc = could not find container \"137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983\": container with ID starting with 137edafd3e00d38a131466463e282574feeea817dc0c572678b50a8077646983 not found: ID does not exist" Jul 10 00:55:54.703804 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.703771 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:55:54.712901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.712867 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:55:54.721515 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:54.716443 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/auth-test-model-d62432-kserve-6fdb4b5f75-qkkx6"] Jul 10 00:55:56.158038 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:56.157997 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09545702-df66-4179-8b9b-4d57e95109b7" path="/var/lib/kubelet/pods/09545702-df66-4179-8b9b-4d57e95109b7/volumes" Jul 10 00:55:59.441330 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:55:59.441285 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:56:15.245328 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:15.245242 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:56:30.536549 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:30.536505 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:56:45.615426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:45.615385 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:56:50.837125 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:50.837082 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:56:51.596901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.596856 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:56:51.597329 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.597313 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09545702-df66-4179-8b9b-4d57e95109b7" containerName="main" Jul 10 00:56:51.597389 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.597331 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="09545702-df66-4179-8b9b-4d57e95109b7" containerName="main" Jul 10 00:56:51.597430 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.597418 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="09545702-df66-4179-8b9b-4d57e95109b7" containerName="main" Jul 10 00:56:51.599537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.599520 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.602117 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.602099 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62432\"/\"rate-test-model-d62432-kserve-self-signed-certs\"" Jul 10 00:56:51.608027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.608003 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:56:51.735142 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735094 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.735333 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735247 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.735333 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735275 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.735333 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735310 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.735470 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735395 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.735470 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.735440 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxtk4\" (UniqueName: \"kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836339 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836306 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxtk4\" (UniqueName: \"kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836357 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836421 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836438 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836465 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836564 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836514 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836874 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836947 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836895 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.836991 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.836947 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.838933 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.838911 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.839366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.839082 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.844149 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.844122 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxtk4\" (UniqueName: \"kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4\") pod \"rate-test-model-d62432-kserve-64f88669cf-b5jqp\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:51.910091 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:51.909990 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:56:52.050531 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:52.050350 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:56:52.053542 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:56:52.053484 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod759601dd_5f0d_4a29_985b_ce74ca051733.slice/crio-60e69c83f9ff091a14ef5fa45acc175673fcae783ec7b47aa4d6d88a1bd99b0f WatchSource:0}: Error finding container 60e69c83f9ff091a14ef5fa45acc175673fcae783ec7b47aa4d6d88a1bd99b0f: Status 404 returned error can't find the container with id 60e69c83f9ff091a14ef5fa45acc175673fcae783ec7b47aa4d6d88a1bd99b0f Jul 10 00:56:52.996957 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:52.996913 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" event={"ID":"759601dd-5f0d-4a29-985b-ce74ca051733","Type":"ContainerStarted","Data":"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919"} Jul 10 00:56:52.996957 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:52.996966 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" event={"ID":"759601dd-5f0d-4a29-985b-ce74ca051733","Type":"ContainerStarted","Data":"60e69c83f9ff091a14ef5fa45acc175673fcae783ec7b47aa4d6d88a1bd99b0f"} Jul 10 00:56:53.014784 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:56:53.014720 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" podStartSLOduration=2.014704831 podStartE2EDuration="2.014704831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:56:53.013115599 +0000 UTC m=+2471.493003151" watchObservedRunningTime="2026-07-10 00:56:53.014704831 +0000 UTC m=+2471.494592396" Jul 10 00:57:00.638504 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:00.638444 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" podUID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" containerName="main" containerID="cri-o://f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384" gracePeriod=45 Jul 10 00:57:00.887919 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:00.887889 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:57:01.029575 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029534 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.029779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029637 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.029779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029657 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.029779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029696 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.029779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029718 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.029779 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029755 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrbjt\" (UniqueName: \"kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt\") pod \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\" (UID: \"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d\") " Jul 10 00:57:01.030055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029852 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:01.030055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029966 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:01.030055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.029980 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home" (OuterVolumeSpecName: "home") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:01.030205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.030137 2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.030205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.030150 2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.030205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.030159 2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.031997 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.031965 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm" (OuterVolumeSpecName: "dshm") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:01.032115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.032017 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:57:01.032115 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.032017 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt" (OuterVolumeSpecName: "kube-api-access-zrbjt") pod "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" (UID: "908d2f62-c65f-4bfa-a8b0-c2b03134ef1d"). InnerVolumeSpecName "kube-api-access-zrbjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:01.047285 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.047255 2548 generic.go:356] "Generic (PLEG): container finished" podID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" containerID="f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384" exitCode=0 Jul 10 00:57:01.047433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.047295 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" Jul 10 00:57:01.047433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.047313 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" event={"ID":"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d","Type":"ContainerDied","Data":"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384"} Jul 10 00:57:01.047433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.047345 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd" event={"ID":"908d2f62-c65f-4bfa-a8b0-c2b03134ef1d","Type":"ContainerDied","Data":"f9e2649996b636a18a3f7f7f997eeb52089baee336d4cd924a640770245074a6"} Jul 10 00:57:01.047433 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.047362 2548 scope.go:117] "RemoveContainer" containerID="f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384" Jul 10 00:57:01.057044 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.057022 2548 scope.go:117] "RemoveContainer" containerID="f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384" Jul 10 00:57:01.057352 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:57:01.057336 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384\": container with ID starting with f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384 not found: ID does not exist" containerID="f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384" Jul 10 00:57:01.057398 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.057361 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384"} err="failed to get container status \"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384\": rpc error: code = NotFound desc = could not find container \"f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384\": container with ID starting with f28d7eb437ec9b8ce70c37997e8ef076bd263d354e41df3aed097cf666063384 not found: ID does not exist" Jul 10 00:57:01.070051 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.070017 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:57:01.071890 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.071870 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/sub-test-model-d62432-kserve-878fddb8f-tcfrd"] Jul 10 00:57:01.131478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.131434 2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.131478 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.131477 2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.131724 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.131486 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrbjt\" (UniqueName: \"kubernetes.io/projected/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d-kube-api-access-zrbjt\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:57:01.911121 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.911075 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:57:01.911121 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.911133 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:57:01.923645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:01.923615 2548 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:57:02.064937 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:02.064903 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:57:02.158379 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:02.158347 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" path="/var/lib/kubelet/pods/908d2f62-c65f-4bfa-a8b0-c2b03134ef1d/volumes" Jul 10 00:57:07.537074 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:07.537029 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:57:24.645869 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:24.645829 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:57:45.040337 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:57:45.040296 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:58:00.248380 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:00.248342 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:58:08.077464 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:08.077421 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:58:09.987175 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.987136 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103572886192"] Jul 10 00:58:09.987594 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.987556 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" containerName="main" Jul 10 00:58:09.987594 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.987569 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" containerName="main" Jul 10 00:58:09.987679 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.987652 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="908d2f62-c65f-4bfa-a8b0-c2b03134ef1d" containerName="main" Jul 10 00:58:09.990787 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.990768 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:09.993548 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.993521 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-v5fdm\"" Jul 10 00:58:09.995607 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:09.995579 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886192"] Jul 10 00:58:10.107249 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.107205 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw86h\" (UniqueName: \"kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h\") pod \"test-curl-7610-140103572886192\" (UID: \"484f989f-24d2-4257-a93d-4df9e8f136c2\") " pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:10.208729 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.208688 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw86h\" (UniqueName: \"kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h\") pod \"test-curl-7610-140103572886192\" (UID: \"484f989f-24d2-4257-a93d-4df9e8f136c2\") " pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:10.216622 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.216586 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw86h\" (UniqueName: \"kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h\") pod \"test-curl-7610-140103572886192\" (UID: \"484f989f-24d2-4257-a93d-4df9e8f136c2\") " pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:10.302138 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.302040 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:10.444844 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.443622 2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:58:10.444844 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:10.443990 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886192"] Jul 10 00:58:11.422354 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:11.422261 2548 generic.go:356] "Generic (PLEG): container finished" podID="484f989f-24d2-4257-a93d-4df9e8f136c2" containerID="c636f74751210c096ddd4c2fd67b3821b78a114230964415c93935b648183baa" exitCode=0 Jul 10 00:58:11.422785 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:11.422346 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103572886192" event={"ID":"484f989f-24d2-4257-a93d-4df9e8f136c2","Type":"ContainerDied","Data":"c636f74751210c096ddd4c2fd67b3821b78a114230964415c93935b648183baa"} Jul 10 00:58:11.422785 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:11.422387 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103572886192" event={"ID":"484f989f-24d2-4257-a93d-4df9e8f136c2","Type":"ContainerStarted","Data":"06481a462c580737167b109fa45994bd54de90d6f288a4877469f68703bcaf5c"} Jul 10 00:58:12.576744 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.576714 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:12.608900 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.608860 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103572886192_484f989f-24d2-4257-a93d-4df9e8f136c2/test-curl-7610-140103572886192/0.log" Jul 10 00:58:12.630275 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.630235 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886192"] Jul 10 00:58:12.632159 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.632137 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886192"] Jul 10 00:58:12.731417 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.731375 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw86h\" (UniqueName: \"kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h\") pod \"484f989f-24d2-4257-a93d-4df9e8f136c2\" (UID: \"484f989f-24d2-4257-a93d-4df9e8f136c2\") " Jul 10 00:58:12.733605 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.733574 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h" (OuterVolumeSpecName: "kube-api-access-dw86h") pod "484f989f-24d2-4257-a93d-4df9e8f136c2" (UID: "484f989f-24d2-4257-a93d-4df9e8f136c2"). InnerVolumeSpecName "kube-api-access-dw86h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:12.761002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.760963 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103572886832"] Jul 10 00:58:12.761426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.761409 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484f989f-24d2-4257-a93d-4df9e8f136c2" containerName="test-curl-7610-140103572886192" Jul 10 00:58:12.761426 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.761426 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="484f989f-24d2-4257-a93d-4df9e8f136c2" containerName="test-curl-7610-140103572886192" Jul 10 00:58:12.761544 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.761510 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="484f989f-24d2-4257-a93d-4df9e8f136c2" containerName="test-curl-7610-140103572886192" Jul 10 00:58:12.764690 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.764659 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:12.772722 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.772688 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886832"] Jul 10 00:58:12.832863 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.832821 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw86h\" (UniqueName: \"kubernetes.io/projected/484f989f-24d2-4257-a93d-4df9e8f136c2-kube-api-access-dw86h\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:12.933745 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.933704 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqd8\" (UniqueName: \"kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8\") pod \"test-curl-7610-140103572886832\" (UID: \"e5fcd45b-09f6-482b-96d4-391526c1d39c\") " pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:12.935964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:12.935938 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:58:13.034810 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.034714 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqd8\" (UniqueName: \"kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8\") pod \"test-curl-7610-140103572886832\" (UID: \"e5fcd45b-09f6-482b-96d4-391526c1d39c\") " pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:13.042394 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.042359 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqd8\" (UniqueName: \"kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8\") pod \"test-curl-7610-140103572886832\" (UID: \"e5fcd45b-09f6-482b-96d4-391526c1d39c\") " pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:13.078548 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.078476 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:13.215459 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.215434 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886832"] Jul 10 00:58:13.218000 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:58:13.217975 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5fcd45b_09f6_482b_96d4_391526c1d39c.slice/crio-b2bdd796bc6df2ab89b0b1feb3a1b3dd824738b713af53110150d373bc2619df WatchSource:0}: Error finding container b2bdd796bc6df2ab89b0b1feb3a1b3dd824738b713af53110150d373bc2619df: Status 404 returned error can't find the container with id b2bdd796bc6df2ab89b0b1feb3a1b3dd824738b713af53110150d373bc2619df Jul 10 00:58:13.434429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.434336 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:13.434429 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.434362 2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06481a462c580737167b109fa45994bd54de90d6f288a4877469f68703bcaf5c" Jul 10 00:58:13.435761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:13.435728 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103572886832" event={"ID":"e5fcd45b-09f6-482b-96d4-391526c1d39c","Type":"ContainerStarted","Data":"b2bdd796bc6df2ab89b0b1feb3a1b3dd824738b713af53110150d373bc2619df"} Jul 10 00:58:13.446204 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:13.446166 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7610-140103572886192\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="484f989f-24d2-4257-a93d-4df9e8f136c2" pod="openshift-ingress/test-curl-7610-140103572886192" Jul 10 00:58:14.158487 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:14.158447 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484f989f-24d2-4257-a93d-4df9e8f136c2" path="/var/lib/kubelet/pods/484f989f-24d2-4257-a93d-4df9e8f136c2/volumes" Jul 10 00:58:14.443612 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:14.443511 2548 generic.go:356] "Generic (PLEG): container finished" podID="e5fcd45b-09f6-482b-96d4-391526c1d39c" containerID="91f13f9327334202e482e108fa5f6026ed4484a2abe4b22e4d0539a2b9acebe4" exitCode=0 Jul 10 00:58:14.443767 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:14.443601 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103572886832" event={"ID":"e5fcd45b-09f6-482b-96d4-391526c1d39c","Type":"ContainerDied","Data":"91f13f9327334202e482e108fa5f6026ed4484a2abe4b22e4d0539a2b9acebe4"} Jul 10 00:58:15.597253 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.597223 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:15.616100 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.616005 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103572886832_e5fcd45b-09f6-482b-96d4-391526c1d39c/test-curl-7610-140103572886832/0.log" Jul 10 00:58:15.638839 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.638798 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886832"] Jul 10 00:58:15.641124 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.641092 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103572886832"] Jul 10 00:58:15.759151 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.759121 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnqd8\" (UniqueName: \"kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8\") pod \"e5fcd45b-09f6-482b-96d4-391526c1d39c\" (UID: \"e5fcd45b-09f6-482b-96d4-391526c1d39c\") " Jul 10 00:58:15.761266 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.761232 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8" (OuterVolumeSpecName: "kube-api-access-pnqd8") pod "e5fcd45b-09f6-482b-96d4-391526c1d39c" (UID: "e5fcd45b-09f6-482b-96d4-391526c1d39c"). InnerVolumeSpecName "kube-api-access-pnqd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:15.804675 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.804636 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:15.805076 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.805062 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fcd45b-09f6-482b-96d4-391526c1d39c" containerName="test-curl-7610-140103572886832" Jul 10 00:58:15.805124 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.805077 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fcd45b-09f6-482b-96d4-391526c1d39c" containerName="test-curl-7610-140103572886832" Jul 10 00:58:15.805160 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.805141 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5fcd45b-09f6-482b-96d4-391526c1d39c" containerName="test-curl-7610-140103572886832" Jul 10 00:58:15.808370 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.808351 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:15.812258 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.812231 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:15.860921 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.860879 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqd8\" (UniqueName: \"kubernetes.io/projected/e5fcd45b-09f6-482b-96d4-391526c1d39c-kube-api-access-pnqd8\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:15.962113 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:15.962071 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvsx\" (UniqueName: \"kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx\") pod \"test-curl-7610-140103577319296\" (UID: \"75371850-b426-4f5d-b72d-a0a247632c4d\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:16.063615 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.063518 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvsx\" (UniqueName: \"kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx\") pod \"test-curl-7610-140103577319296\" (UID: \"75371850-b426-4f5d-b72d-a0a247632c4d\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:16.072666 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.072637 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvsx\" (UniqueName: \"kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx\") pod \"test-curl-7610-140103577319296\" (UID: \"75371850-b426-4f5d-b72d-a0a247632c4d\") " pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:16.118708 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.118650 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:16.158528 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.158473 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5fcd45b-09f6-482b-96d4-391526c1d39c" path="/var/lib/kubelet/pods/e5fcd45b-09f6-482b-96d4-391526c1d39c/volumes" Jul 10 00:58:16.266211 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.266176 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:16.269289 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:58:16.269246 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75371850_b426_4f5d_b72d_a0a247632c4d.slice/crio-9fbb493773025b0c06f61a1c57bcc84b4d99e14586f36c3759f8ac7e91849f06 WatchSource:0}: Error finding container 9fbb493773025b0c06f61a1c57bcc84b4d99e14586f36c3759f8ac7e91849f06: Status 404 returned error can't find the container with id 9fbb493773025b0c06f61a1c57bcc84b4d99e14586f36c3759f8ac7e91849f06 Jul 10 00:58:16.455128 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.455098 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103572886832" Jul 10 00:58:16.455336 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.455134 2548 scope.go:117] "RemoveContainer" containerID="91f13f9327334202e482e108fa5f6026ed4484a2abe4b22e4d0539a2b9acebe4" Jul 10 00:58:16.456546 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:16.456342 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577319296" event={"ID":"75371850-b426-4f5d-b72d-a0a247632c4d","Type":"ContainerStarted","Data":"9fbb493773025b0c06f61a1c57bcc84b4d99e14586f36c3759f8ac7e91849f06"} Jul 10 00:58:17.462937 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:17.462902 2548 generic.go:356] "Generic (PLEG): container finished" podID="75371850-b426-4f5d-b72d-a0a247632c4d" containerID="3348f43c207459b6f55e1eed736aed5613aa3c8cc45ebcfa7ba12778cdd06ef7" exitCode=0 Jul 10 00:58:17.463396 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:17.462986 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577319296" event={"ID":"75371850-b426-4f5d-b72d-a0a247632c4d","Type":"ContainerDied","Data":"3348f43c207459b6f55e1eed736aed5613aa3c8cc45ebcfa7ba12778cdd06ef7"} Jul 10 00:58:18.625029 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.625002 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:18.643055 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.643025 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103577319296_75371850-b426-4f5d-b72d-a0a247632c4d/test-curl-7610-140103577319296/0.log" Jul 10 00:58:18.666101 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.666061 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:18.667952 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.667930 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103577319296"] Jul 10 00:58:18.791528 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.791399 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxvsx\" (UniqueName: \"kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx\") pod \"75371850-b426-4f5d-b72d-a0a247632c4d\" (UID: \"75371850-b426-4f5d-b72d-a0a247632c4d\") " Jul 10 00:58:18.793657 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.793627 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx" (OuterVolumeSpecName: "kube-api-access-zxvsx") pod "75371850-b426-4f5d-b72d-a0a247632c4d" (UID: "75371850-b426-4f5d-b72d-a0a247632c4d"). InnerVolumeSpecName "kube-api-access-zxvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:18.892997 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:18.892953 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvsx\" (UniqueName: \"kubernetes.io/projected/75371850-b426-4f5d-b72d-a0a247632c4d-kube-api-access-zxvsx\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:19.475846 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:19.475809 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:19.478034 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:19.475813 2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fbb493773025b0c06f61a1c57bcc84b4d99e14586f36c3759f8ac7e91849f06" Jul 10 00:58:19.486424 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:19.486385 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7610-140103577319296\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="75371850-b426-4f5d-b72d-a0a247632c4d" pod="openshift-ingress/test-curl-7610-140103577319296" Jul 10 00:58:20.158542 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:20.158482 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75371850-b426-4f5d-b72d-a0a247632c4d" path="/var/lib/kubelet/pods/75371850-b426-4f5d-b72d-a0a247632c4d/volumes" Jul 10 00:58:23.098941 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.098894 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" podUID="759601dd-5f0d-4a29-985b-ce74ca051733" containerName="main" containerID="cri-o://8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919" gracePeriod=45 Jul 10 00:58:23.371400 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.371370 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:58:23.441366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441323 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441390 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxtk4\" (UniqueName: \"kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441439 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441470 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441520 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441585 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441553 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache\") pod \"759601dd-5f0d-4a29-985b-ce74ca051733\" (UID: \"759601dd-5f0d-4a29-985b-ce74ca051733\") " Jul 10 00:58:23.441862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441762 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:23.441862 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441771 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home" (OuterVolumeSpecName: "home") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:23.441969 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.441902 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache" (OuterVolumeSpecName: "model-cache") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:23.443647 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.443616 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm" (OuterVolumeSpecName: "dshm") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:23.443647 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.443629 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:58:23.443799 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.443651 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4" (OuterVolumeSpecName: "kube-api-access-dxtk4") pod "759601dd-5f0d-4a29-985b-ce74ca051733" (UID: "759601dd-5f0d-4a29-985b-ce74ca051733"). InnerVolumeSpecName "kube-api-access-dxtk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:23.496175 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.496141 2548 generic.go:356] "Generic (PLEG): container finished" podID="759601dd-5f0d-4a29-985b-ce74ca051733" containerID="8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919" exitCode=0 Jul 10 00:58:23.496374 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.496184 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" event={"ID":"759601dd-5f0d-4a29-985b-ce74ca051733","Type":"ContainerDied","Data":"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919"} Jul 10 00:58:23.496374 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.496198 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" Jul 10 00:58:23.496374 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.496216 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp" event={"ID":"759601dd-5f0d-4a29-985b-ce74ca051733","Type":"ContainerDied","Data":"60e69c83f9ff091a14ef5fa45acc175673fcae783ec7b47aa4d6d88a1bd99b0f"} Jul 10 00:58:23.496374 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.496233 2548 scope.go:117] "RemoveContainer" containerID="8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919" Jul 10 00:58:23.506568 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.506477 2548 scope.go:117] "RemoveContainer" containerID="8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919" Jul 10 00:58:23.506880 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:23.506852 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919\": container with ID starting with 8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919 not found: ID does not exist" containerID="8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919" Jul 10 00:58:23.506972 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.506888 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919"} err="failed to get container status \"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919\": rpc error: code = NotFound desc = could not find container \"8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919\": container with ID starting with 8895e3fbe20f6376f5d1618b0baf77d19224980f195de76af26968f56da15919 not found: ID does not exist" Jul 10 00:58:23.519700 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.519663 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:58:23.522931 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.522897 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d62432/rate-test-model-d62432-kserve-64f88669cf-b5jqp"] Jul 10 00:58:23.542453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542411 2548 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:23.542453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542449 2548 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:23.542453 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542459 2548 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759601dd-5f0d-4a29-985b-ce74ca051733-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:23.542713 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542469 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxtk4\" (UniqueName: \"kubernetes.io/projected/759601dd-5f0d-4a29-985b-ce74ca051733-kube-api-access-dxtk4\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:23.542713 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542481 2548 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:23.542713 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:23.542521 2548 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759601dd-5f0d-4a29-985b-ce74ca051733-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:24.158436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.158401 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759601dd-5f0d-4a29-985b-ce74ca051733" path="/var/lib/kubelet/pods/759601dd-5f0d-4a29-985b-ce74ca051733/volumes" Jul 10 00:58:24.590278 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590241 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:24.590687 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590675 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759601dd-5f0d-4a29-985b-ce74ca051733" containerName="main" Jul 10 00:58:24.590737 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590690 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="759601dd-5f0d-4a29-985b-ce74ca051733" containerName="main" Jul 10 00:58:24.590737 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590708 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75371850-b426-4f5d-b72d-a0a247632c4d" containerName="test-curl-7610-140103577319296" Jul 10 00:58:24.590737 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590715 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="75371850-b426-4f5d-b72d-a0a247632c4d" containerName="test-curl-7610-140103577319296" Jul 10 00:58:24.590840 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590781 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="75371850-b426-4f5d-b72d-a0a247632c4d" containerName="test-curl-7610-140103577319296" Jul 10 00:58:24.590840 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.590793 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="759601dd-5f0d-4a29-985b-ce74ca051733" containerName="main" Jul 10 00:58:24.595402 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.595378 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:24.597830 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.597800 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:24.598101 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.598077 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-v5fdm\"" Jul 10 00:58:24.651867 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.651825 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtlc9\" (UniqueName: \"kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9\") pod \"test-curl-7610-140103577320304\" (UID: \"ff9791b0-25ab-430c-8c24-6395c655b9e7\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:24.752859 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.752822 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtlc9\" (UniqueName: \"kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9\") pod \"test-curl-7610-140103577320304\" (UID: \"ff9791b0-25ab-430c-8c24-6395c655b9e7\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:24.761726 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.761692 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtlc9\" (UniqueName: \"kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9\") pod \"test-curl-7610-140103577320304\" (UID: \"ff9791b0-25ab-430c-8c24-6395c655b9e7\") " pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:24.905455 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:24.905360 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:25.054902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:25.054869 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:25.056746 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:58:25.056716 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff9791b0_25ab_430c_8c24_6395c655b9e7.slice/crio-e429ad042c62e4ffff57eb069dfae7cad8d82b5205ec68f231c59bce32ee83d0 WatchSource:0}: Error finding container e429ad042c62e4ffff57eb069dfae7cad8d82b5205ec68f231c59bce32ee83d0: Status 404 returned error can't find the container with id e429ad042c62e4ffff57eb069dfae7cad8d82b5205ec68f231c59bce32ee83d0 Jul 10 00:58:25.509445 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:25.509402 2548 generic.go:356] "Generic (PLEG): container finished" podID="ff9791b0-25ab-430c-8c24-6395c655b9e7" containerID="cab1ba00d925be8cf4702caddc6b904add8ec49a4d05d8d0ea5c7ebc39e0d0ee" exitCode=0 Jul 10 00:58:25.509883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:25.509483 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577320304" event={"ID":"ff9791b0-25ab-430c-8c24-6395c655b9e7","Type":"ContainerDied","Data":"cab1ba00d925be8cf4702caddc6b904add8ec49a4d05d8d0ea5c7ebc39e0d0ee"} Jul 10 00:58:25.509883 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:25.509563 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7610-140103577320304" event={"ID":"ff9791b0-25ab-430c-8c24-6395c655b9e7","Type":"ContainerStarted","Data":"e429ad042c62e4ffff57eb069dfae7cad8d82b5205ec68f231c59bce32ee83d0"} Jul 10 00:58:26.666209 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.666180 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:26.684878 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.684839 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7610-140103577320304_ff9791b0-25ab-430c-8c24-6395c655b9e7/test-curl-7610-140103577320304/0.log" Jul 10 00:58:26.708260 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.708216 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:26.711770 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.711739 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7610-140103577320304"] Jul 10 00:58:26.771214 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.771121 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtlc9\" (UniqueName: \"kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9\") pod \"ff9791b0-25ab-430c-8c24-6395c655b9e7\" (UID: \"ff9791b0-25ab-430c-8c24-6395c655b9e7\") " Jul 10 00:58:26.773229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.773202 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9" (OuterVolumeSpecName: "kube-api-access-qtlc9") pod "ff9791b0-25ab-430c-8c24-6395c655b9e7" (UID: "ff9791b0-25ab-430c-8c24-6395c655b9e7"). InnerVolumeSpecName "kube-api-access-qtlc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:26.871972 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:26.871939 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtlc9\" (UniqueName: \"kubernetes.io/projected/ff9791b0-25ab-430c-8c24-6395c655b9e7-kube-api-access-qtlc9\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:27.522219 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:27.522182 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:27.522403 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:27.522226 2548 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e429ad042c62e4ffff57eb069dfae7cad8d82b5205ec68f231c59bce32ee83d0" Jul 10 00:58:27.532670 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:27.532639 2548 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7610-140103577320304\" is forbidden: User \"system:node:ip-10-0-137-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-8.ec2.internal' and this object" podUID="ff9791b0-25ab-430c-8c24-6395c655b9e7" pod="openshift-ingress/test-curl-7610-140103577320304" Jul 10 00:58:28.158134 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:28.158094 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff9791b0-25ab-430c-8c24-6395c655b9e7" path="/var/lib/kubelet/pods/ff9791b0-25ab-430c-8c24-6395c655b9e7/volumes" Jul 10 00:58:30.041940 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:30.041903 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:58:33.547706 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:33.547663 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:58:33.548223 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:33.547918 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" containerID="cri-o://b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e" gracePeriod=30 Jul 10 00:58:33.736835 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:33.736793 2548 patch_prober.go:28] interesting pod/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 00:58:33.737025 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:33.736855 2548 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:58:38.810572 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.810465 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:58:38.892517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892454 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892526 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892638 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892684 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892719 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892744 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.892793 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892772 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.893048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892806 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.893048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892804 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:38.893048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892875 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:38.893048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892943 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz5mx\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx\") pod \"8ab236d6-fc03-48f5-9e66-da6f0b807060\" (UID: \"8ab236d6-fc03-48f5-9e66-da6f0b807060\") " Jul 10 00:58:38.893048 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.892978 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:38.893297 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893200 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:58:38.893297 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893289 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.893381 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893309 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.893381 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893322 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab236d6-fc03-48f5-9e66-da6f0b807060-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.893381 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893336 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.893910 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.893878 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data" (OuterVolumeSpecName: "istio-data") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:38.894964 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.894944 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:38.895138 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.895114 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx" (OuterVolumeSpecName: "kube-api-access-bz5mx") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "kube-api-access-bz5mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:38.895351 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.895326 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:58:38.895351 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.895339 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token" (OuterVolumeSpecName: "istio-token") pod "8ab236d6-fc03-48f5-9e66-da6f0b807060" (UID: "8ab236d6-fc03-48f5-9e66-da6f0b807060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:38.994872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.994817 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.994872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.994864 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.994872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.994877 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.994872 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.994888 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab236d6-fc03-48f5-9e66-da6f0b807060-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:38.995160 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:38.994901 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5mx\" (UniqueName: \"kubernetes.io/projected/8ab236d6-fc03-48f5-9e66-da6f0b807060-kube-api-access-bz5mx\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:39.594286 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.594250 2548 generic.go:356] "Generic (PLEG): container finished" podID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerID="b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e" exitCode=0 Jul 10 00:58:39.594540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.594322 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" Jul 10 00:58:39.594540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.594335 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerDied","Data":"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e"} Jul 10 00:58:39.594540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.594431 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9" event={"ID":"8ab236d6-fc03-48f5-9e66-da6f0b807060","Type":"ContainerDied","Data":"ad819a0809b46f1da6ee4a396e7fa65892731803bb2646099fb4bbef6cc804de"} Jul 10 00:58:39.594540 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.594461 2548 scope.go:117] "RemoveContainer" containerID="b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e" Jul 10 00:58:39.604029 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.604004 2548 scope.go:117] "RemoveContainer" containerID="c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e" Jul 10 00:58:39.612584 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.612557 2548 scope.go:117] "RemoveContainer" containerID="b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e" Jul 10 00:58:39.612895 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:39.612875 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e\": container with ID starting with b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e not found: ID does not exist" containerID="b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e" Jul 10 00:58:39.613007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.612905 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e"} err="failed to get container status \"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e\": rpc error: code = NotFound desc = could not find container \"b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e\": container with ID starting with b225c00c7042da5d1ca275f62369e64b010a42967358645de7a8d2c580a9da5e not found: ID does not exist" Jul 10 00:58:39.613007 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.612984 2548 scope.go:117] "RemoveContainer" containerID="c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e" Jul 10 00:58:39.613290 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:39.613265 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e\": container with ID starting with c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e not found: ID does not exist" containerID="c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e" Jul 10 00:58:39.613364 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.613298 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e"} err="failed to get container status \"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e\": rpc error: code = NotFound desc = could not find container \"c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e\": container with ID starting with c4b6b0b352c2fdae46e7d2bd02a33172973bc0e919cd8864f20060e001176f2e not found: ID does not exist" Jul 10 00:58:39.616154 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.616131 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:58:39.621516 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:39.621475 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d62432-openshift-default-7b4b7b4487-m7dv9"] Jul 10 00:58:40.158058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:40.158020 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" path="/var/lib/kubelet/pods/8ab236d6-fc03-48f5-9e66-da6f0b807060/volumes" Jul 10 00:58:40.333263 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:40.333220 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:58:40.333578 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:40.333540 2548 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" containerID="cri-o://75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d" gracePeriod=30 Jul 10 00:58:43.439438 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:43.439395 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2sqfl"] Jul 10 00:58:45.601376 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.601302 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:58:45.633775 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.633733 2548 generic.go:356] "Generic (PLEG): container finished" podID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerID="75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d" exitCode=0 Jul 10 00:58:45.633946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.633780 2548 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" Jul 10 00:58:45.633946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.633788 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerDied","Data":"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d"} Jul 10 00:58:45.633946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.633844 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7" event={"ID":"ce4510c2-873d-4fa2-ba71-d9a7bfda555c","Type":"ContainerDied","Data":"f8ecae8fbb807d789fe564445297393572426e3fd02e2c2aa2170e7ec0424206"} Jul 10 00:58:45.633946 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.633876 2548 scope.go:117] "RemoveContainer" containerID="75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d" Jul 10 00:58:45.644411 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.644391 2548 scope.go:117] "RemoveContainer" containerID="e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661" Jul 10 00:58:45.652645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.652622 2548 scope.go:117] "RemoveContainer" containerID="75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d" Jul 10 00:58:45.652974 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:45.652948 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d\": container with ID starting with 75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d not found: ID does not exist" containerID="75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d" Jul 10 00:58:45.653027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.652991 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d"} err="failed to get container status \"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d\": rpc error: code = NotFound desc = could not find container \"75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d\": container with ID starting with 75ed65ded4d540c5e176a58dc38302b9fbc061427e34134b61ce252e8988921d not found: ID does not exist" Jul 10 00:58:45.653027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.653016 2548 scope.go:117] "RemoveContainer" containerID="e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661" Jul 10 00:58:45.653314 ip-10-0-137-8 kubenswrapper[2548]: E0710 00:58:45.653286 2548 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661\": container with ID starting with e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661 not found: ID does not exist" containerID="e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661" Jul 10 00:58:45.653386 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.653321 2548 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661"} err="failed to get container status \"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661\": rpc error: code = NotFound desc = could not find container \"e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661\": container with ID starting with e7399673400e9ea4f764840c7c3865d78429edb14114df9be2a6855fa50cc661 not found: ID does not exist" Jul 10 00:58:45.758538 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758454 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.758731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758554 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.758731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758595 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.758731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758624 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.758731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758657 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4gt9\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.758731 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758713 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.759002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758740 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.759002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758783 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.759002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758813 2548 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token\") pod \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\" (UID: \"ce4510c2-873d-4fa2-ba71-d9a7bfda555c\") " Jul 10 00:58:45.759002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.758868 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:45.759205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759046 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:45.759205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759119 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:45.759205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759177 2548 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-credential-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.759205 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759195 2548 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-socket\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.759379 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759318 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:58:45.759520 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.759473 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:45.761097 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.761064 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9" (OuterVolumeSpecName: "kube-api-access-l4gt9") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "kube-api-access-l4gt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:45.761217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.761101 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:58:45.761217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.761100 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:45.761217 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.761118 2548 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce4510c2-873d-4fa2-ba71-d9a7bfda555c" (UID: "ce4510c2-873d-4fa2-ba71-d9a7bfda555c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860682 2548 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-data\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860709 2548 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istiod-ca-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860719 2548 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4gt9\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-kube-api-access-l4gt9\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860728 2548 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-podinfo\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860738 2548 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-envoy\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860746 2548 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-workload-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.860774 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.860754 2548 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce4510c2-873d-4fa2-ba71-d9a7bfda555c-istio-token\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 10 00:58:45.956524 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.956470 2548 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:58:45.961609 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:45.961577 2548 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f84011-openshift-default-5fc7bd8967-2gbw7"] Jul 10 00:58:46.012210 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.012170 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ck4wd_0935a234-7300-4876-b8ba-ea0465781d6c/manager/0.log" Jul 10 00:58:46.140272 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.140182 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xdtpt_f91ee870-b4fc-4a83-8196-a43a7fefae05/manager/0.log" Jul 10 00:58:46.158002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.157964 2548 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" path="/var/lib/kubelet/pods/ce4510c2-873d-4fa2-ba71-d9a7bfda555c/volumes" Jul 10 00:58:46.272748 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.272717 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5687559b5f-vkflj_d92b957f-1338-471f-be8f-c35744d84181/manager/0.log" Jul 10 00:58:46.518442 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.518403 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-szt7w_0e1047a7-937f-4749-8f28-e09a9b13e5b6/manager/0.log" Jul 10 00:58:46.634464 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.634425 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-bfhdp_10514622-9736-4be7-9c30-c44590b111c0/copy-manifests/0.log" Jul 10 00:58:46.641093 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:46.641064 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-bfhdp_10514622-9736-4be7-9c30-c44590b111c0/manager/0.log" Jul 10 00:58:47.378277 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:47.378247 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5dh75_fde59454-4851-4a42-a4b0-917c2f075c5e/postgres/0.log" Jul 10 00:58:48.849327 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:48.849288 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-c2rgg_2bce3542-f022-4c5e-bad5-08241d8fd951/manager/0.log" Jul 10 00:58:48.962901 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:48.962867 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tfsl8_c1fb7182-0bfe-467f-a2e6-88267a309b90/manager/0.log" Jul 10 00:58:49.075772 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:49.075743 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xvpnh_a8626e72-64c9-42ad-a97c-95ed2502c95d/kuadrant-console-plugin/0.log" Jul 10 00:58:49.205436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:49.205409 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wdxsz_eb9e5721-2292-4990-b0fe-560c698c8de8/registry-server/0.log" Jul 10 00:58:49.462010 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:49.461985 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2sqfl_75634734-f19d-4a59-b49f-4483f56c2dd0/limitador/0.log" Jul 10 00:58:49.592510 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:49.592454 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t9sf8_25b9d0dd-77df-4c15-9c14-d27ae4212d6d/manager/0.log" Jul 10 00:58:50.198517 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:50.198468 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-879s2_1de655c6-c63a-4ee0-acd6-68f404f95f8e/kube-auth-proxy/0.log" Jul 10 00:58:50.708915 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:50.708876 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pnsbg_af03b31b-075b-4e7f-a352-29cef2d1a77c/payload-processing/0.log" Jul 10 00:58:51.189877 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:51.189795 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-jvkzd_2f592778-ad5e-4900-859b-b349dc7675c8/main/0.log" Jul 10 00:58:51.544659 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:51.544622 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwhc2b_dc958413-0dbb-413d-8289-575f3066a5bb/main/0.log" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.534108 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcnxx/must-gather-nwp92"] Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.534964 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.534988 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535007 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff9791b0-25ab-430c-8c24-6395c655b9e7" containerName="test-curl-7610-140103577320304" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535017 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff9791b0-25ab-430c-8c24-6395c655b9e7" containerName="test-curl-7610-140103577320304" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535044 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535054 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535064 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535073 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535089 2548 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535098 2548 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535272 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff9791b0-25ab-430c-8c24-6395c655b9e7" containerName="test-curl-7610-140103577320304" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535290 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535303 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535319 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce4510c2-873d-4fa2-ba71-d9a7bfda555c" containerName="istio-proxy" Jul 10 00:58:55.537089 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.535677 2548 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab236d6-fc03-48f5-9e66-da6f0b807060" containerName="istio-proxy" Jul 10 00:58:55.540078 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.540057 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.542555 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.542488 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcnxx\"/\"openshift-service-ca.crt\"" Jul 10 00:58:55.542555 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.542537 2548 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcnxx\"/\"default-dockercfg-cv5kp\"" Jul 10 00:58:55.543229 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.543206 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/must-gather-nwp92"] Jul 10 00:58:55.544066 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.543858 2548 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcnxx\"/\"kube-root-ca.crt\"" Jul 10 00:58:55.648539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.648475 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jzl\" (UniqueName: \"kubernetes.io/projected/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-kube-api-access-47jzl\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.648539 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.648541 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-must-gather-output\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.750001 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.749961 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-must-gather-output\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.750224 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.750116 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jzl\" (UniqueName: \"kubernetes.io/projected/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-kube-api-access-47jzl\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.750319 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.750298 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-must-gather-output\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.758428 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.758397 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jzl\" (UniqueName: \"kubernetes.io/projected/15fdcd8e-d4f5-4e08-9b0c-c552c5459f50-kube-api-access-47jzl\") pod \"must-gather-nwp92\" (UID: \"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50\") " pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.851032 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.850940 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/must-gather-nwp92" Jul 10 00:58:55.990013 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:55.989983 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/must-gather-nwp92"] Jul 10 00:58:55.992635 ip-10-0-137-8 kubenswrapper[2548]: W0710 00:58:55.992590 2548 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15fdcd8e_d4f5_4e08_9b0c_c552c5459f50.slice/crio-28fdf62b8768bfdebfdae61348205360d12a6c6c7e79a1be2eebfd11a266d908 WatchSource:0}: Error finding container 28fdf62b8768bfdebfdae61348205360d12a6c6c7e79a1be2eebfd11a266d908: Status 404 returned error can't find the container with id 28fdf62b8768bfdebfdae61348205360d12a6c6c7e79a1be2eebfd11a266d908 Jul 10 00:58:56.691880 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:56.691854 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/must-gather-nwp92" event={"ID":"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50","Type":"ContainerStarted","Data":"28fdf62b8768bfdebfdae61348205360d12a6c6c7e79a1be2eebfd11a266d908"} Jul 10 00:58:57.701912 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:57.701865 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/must-gather-nwp92" event={"ID":"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50","Type":"ContainerStarted","Data":"cdd9799360f188edfc6cc9dcc84d8de9860894b4bae03d7988745bcea98011ef"} Jul 10 00:58:57.702376 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:57.701930 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/must-gather-nwp92" event={"ID":"15fdcd8e-d4f5-4e08-9b0c-c552c5459f50","Type":"ContainerStarted","Data":"ab9884677543ea8b611028bc8b3e96b6fbaa24dfa00ddc0e2554aa578ccc28ef"} Jul 10 00:58:57.719681 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:57.719623 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcnxx/must-gather-nwp92" podStartSLOduration=2.023567074 podStartE2EDuration="2.719604233s" totalImagesPullingTime="696.037159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:58:57.717023337 +0000 UTC m=+2596.196910887" watchObservedRunningTime="2026-07-10 00:58:57.719604233 +0000 UTC m=+2596.199491781" Jul 10 00:58:58.185765 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:58.185672 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mf9qj_4921e36d-183c-4039-a49d-56ede70a834c/global-pull-secret-syncer/0.log" Jul 10 00:58:58.295948 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:58.295903 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h4dsd_ed4cf4c8-026c-40d4-b0ab-18b4f6e404b4/konnectivity-agent/0.log" Jul 10 00:58:58.398129 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:58:58.398094 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-8.ec2.internal_c365a80243b37665c2c8c1b5f01aa50a/haproxy/0.log" Jul 10 00:59:03.157154 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.157115 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-c2rgg_2bce3542-f022-4c5e-bad5-08241d8fd951/manager/0.log" Jul 10 00:59:03.180723 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.180685 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-tfsl8_c1fb7182-0bfe-467f-a2e6-88267a309b90/manager/0.log" Jul 10 00:59:03.219637 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.219604 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-xvpnh_a8626e72-64c9-42ad-a97c-95ed2502c95d/kuadrant-console-plugin/0.log" Jul 10 00:59:03.252002 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.251961 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wdxsz_eb9e5721-2292-4990-b0fe-560c698c8de8/registry-server/0.log" Jul 10 00:59:03.373755 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.373730 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2sqfl_75634734-f19d-4a59-b49f-4483f56c2dd0/limitador/0.log" Jul 10 00:59:03.457938 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:03.457898 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-t9sf8_25b9d0dd-77df-4c15-9c14-d27ae4212d6d/manager/0.log" Jul 10 00:59:05.112027 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.111937 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/alertmanager/0.log" Jul 10 00:59:05.130424 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.130382 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/config-reloader/0.log" Jul 10 00:59:05.148678 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.148647 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/kube-rbac-proxy-web/0.log" Jul 10 00:59:05.168665 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.168576 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/kube-rbac-proxy/0.log" Jul 10 00:59:05.189640 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.189606 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/kube-rbac-proxy-metric/0.log" Jul 10 00:59:05.208440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.208265 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/prom-label-proxy/0.log" Jul 10 00:59:05.228341 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.228240 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03208ada-d175-4176-932a-901db8659f1f/init-config-reloader/0.log" Jul 10 00:59:05.295041 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.294968 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-n66hc_b60a2f87-d93a-435a-b087-0bbeccb24351/kube-state-metrics/0.log" Jul 10 00:59:05.309034 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.309006 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-n66hc_b60a2f87-d93a-435a-b087-0bbeccb24351/kube-rbac-proxy-main/0.log" Jul 10 00:59:05.326455 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.326423 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-n66hc_b60a2f87-d93a-435a-b087-0bbeccb24351/kube-rbac-proxy-self/0.log" Jul 10 00:59:05.482084 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.482057 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrc5s_bb70b152-4671-4cf1-bcb7-956cc02bd0f8/node-exporter/0.log" Jul 10 00:59:05.500440 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.500409 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrc5s_bb70b152-4671-4cf1-bcb7-956cc02bd0f8/kube-rbac-proxy/0.log" Jul 10 00:59:05.517373 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.517342 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nrc5s_bb70b152-4671-4cf1-bcb7-956cc02bd0f8/init-textfile/0.log" Jul 10 00:59:05.956476 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.956445 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-f8mdm_6bc04754-ff85-41be-91e8-c2bc59cc4b98/prometheus-operator-admission-webhook/0.log" Jul 10 00:59:05.981817 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.981789 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f87c7cbff-hsgqm_427040be-9b5c-4ae6-a1c2-254378bd7371/telemeter-client/0.log" Jul 10 00:59:05.998916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:05.998889 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f87c7cbff-hsgqm_427040be-9b5c-4ae6-a1c2-254378bd7371/reload/0.log" Jul 10 00:59:06.020407 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:06.020359 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f87c7cbff-hsgqm_427040be-9b5c-4ae6-a1c2-254378bd7371/kube-rbac-proxy/0.log" Jul 10 00:59:08.742802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:08.742768 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78779dd596-frwdc_573cd238-679c-4793-b69b-79ede849ab12/console/0.log" Jul 10 00:59:09.401685 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.401643 2548 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj"] Jul 10 00:59:09.410186 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.410149 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.415719 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.415566 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj"] Jul 10 00:59:09.503366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.503001 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-lib-modules\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.503366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.503050 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-sys\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.503366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.503083 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7c9\" (UniqueName: \"kubernetes.io/projected/e6abfcdb-99ef-4f52-af51-7438c233c353-kube-api-access-qw7c9\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.503366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.503218 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-podres\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.503366 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.503248 2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-proc\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604593 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-podres\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604644 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-proc\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604686 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-lib-modules\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604701 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-sys\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604719 2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7c9\" (UniqueName: \"kubernetes.io/projected/e6abfcdb-99ef-4f52-af51-7438c233c353-kube-api-access-qw7c9\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604770 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-podres\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604790 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-proc\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604813 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-sys\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.604884 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.604831 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6abfcdb-99ef-4f52-af51-7438c233c353-lib-modules\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.612645 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.612608 2548 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7c9\" (UniqueName: \"kubernetes.io/projected/e6abfcdb-99ef-4f52-af51-7438c233c353-kube-api-access-qw7c9\") pod \"perf-node-gather-daemonset-5rzcj\" (UID: \"e6abfcdb-99ef-4f52-af51-7438c233c353\") " pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.730709 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.730667 2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:09.924944 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:09.924915 2548 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj"] Jul 10 00:59:10.580383 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.580348 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5hkd_0559a2bd-a537-451d-ba80-a6a74fbf2067/dns/0.log" Jul 10 00:59:10.596679 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.596649 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5hkd_0559a2bd-a537-451d-ba80-a6a74fbf2067/kube-rbac-proxy/0.log" Jul 10 00:59:10.642769 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.642741 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9qxd8_e9015bc2-ec2b-4800-b911-1e190522f37e/dns-node-resolver/0.log" Jul 10 00:59:10.804729 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.804680 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" event={"ID":"e6abfcdb-99ef-4f52-af51-7438c233c353","Type":"ContainerStarted","Data":"6c4fd66a2241fa521504c06c7999ea36f55c00da95d6ae3d596eb3a0b6d0bed5"} Jul 10 00:59:10.804916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.804794 2548 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" event={"ID":"e6abfcdb-99ef-4f52-af51-7438c233c353","Type":"ContainerStarted","Data":"99f86c12c665f06d7e835c6d2bf3ca8c36c18e174f3beb025b40d42132d74a88"} Jul 10 00:59:10.804916 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.804827 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:10.820974 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:10.820911 2548 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" podStartSLOduration=1.820889706 podStartE2EDuration="1.820889706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:59:10.81978169 +0000 UTC m=+2609.299669262" watchObservedRunningTime="2026-07-10 00:59:10.820889706 +0000 UTC m=+2609.300777254" Jul 10 00:59:11.159077 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:11.159040 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cf9f5cbff-drn4g_0d58d34c-7595-420a-a663-238ff022a889/registry/0.log" Jul 10 00:59:11.178449 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:11.178409 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6cf9f5cbff-drn4g_0d58d34c-7595-420a-a663-238ff022a889/registry/1.log" Jul 10 00:59:11.223282 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:11.223244 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkxgp_f80f9e53-8206-4724-9677-d1c46b2887f4/node-ca/0.log" Jul 10 00:59:12.287477 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:12.287450 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-879s2_1de655c6-c63a-4ee0-acd6-68f404f95f8e/kube-auth-proxy/0.log" Jul 10 00:59:12.451163 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:12.451131 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pnsbg_af03b31b-075b-4e7f-a352-29cef2d1a77c/payload-processing/0.log" Jul 10 00:59:13.140740 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:13.140706 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7ssjm_5c7a8fbe-0641-4a83-91aa-28ce36b25468/serve-healthcheck-canary/0.log" Jul 10 00:59:13.641935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:13.641908 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5tqf_f0d21b32-6172-4399-889c-ea552b902408/kube-rbac-proxy/0.log" Jul 10 00:59:13.659682 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:13.659651 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5tqf_f0d21b32-6172-4399-889c-ea552b902408/exporter/0.log" Jul 10 00:59:13.678511 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:13.678472 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b5tqf_f0d21b32-6172-4399-889c-ea552b902408/extractor/0.log" Jul 10 00:59:15.868802 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:15.868763 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ck4wd_0935a234-7300-4876-b8ba-ea0465781d6c/manager/0.log" Jul 10 00:59:16.121066 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.120967 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xdtpt_f91ee870-b4fc-4a83-8196-a43a7fefae05/manager/0.log" Jul 10 00:59:16.289218 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.289183 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5687559b5f-vkflj_d92b957f-1338-471f-be8f-c35744d84181/manager/0.log" Jul 10 00:59:16.413336 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.413242 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-szt7w_0e1047a7-937f-4749-8f28-e09a9b13e5b6/manager/0.log" Jul 10 00:59:16.433902 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.433873 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-bfhdp_10514622-9736-4be7-9c30-c44590b111c0/manager/0.log" Jul 10 00:59:16.451510 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.451467 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559fc4999-bfhdp_10514622-9736-4be7-9c30-c44590b111c0/copy-manifests/0.log" Jul 10 00:59:16.825047 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:16.825016 2548 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcnxx/perf-node-gather-daemonset-5rzcj" Jul 10 00:59:19.576906 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:19.576867 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-879s2_1de655c6-c63a-4ee0-acd6-68f404f95f8e/kube-auth-proxy/0.log" Jul 10 00:59:19.712580 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:19.712540 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pnsbg_af03b31b-075b-4e7f-a352-29cef2d1a77c/payload-processing/0.log" Jul 10 00:59:21.083346 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:21.083310 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b9f4c495f-879s2_1de655c6-c63a-4ee0-acd6-68f404f95f8e/kube-auth-proxy/0.log" Jul 10 00:59:21.228698 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:21.228669 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-pnsbg_af03b31b-075b-4e7f-a352-29cef2d1a77c/payload-processing/0.log" Jul 10 00:59:24.511196 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.511161 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8c2g2_3dcab713-d3de-40c9-9c69-bb98ce4d93f0/kube-multus/0.log" Jul 10 00:59:24.662311 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.662283 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/kube-multus-additional-cni-plugins/0.log" Jul 10 00:59:24.677761 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.677733 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/egress-router-binary-copy/0.log" Jul 10 00:59:24.693319 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.693286 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/cni-plugins/0.log" Jul 10 00:59:24.708616 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.708580 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/bond-cni-plugin/0.log" Jul 10 00:59:24.723721 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.723692 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/routeoverride-cni/0.log" Jul 10 00:59:24.738935 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.738907 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/whereabouts-cni-bincopy/0.log" Jul 10 00:59:24.754436 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.754410 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8md9r_482f31e5-175d-4782-8d99-e51748a8b929/whereabouts-cni/0.log" Jul 10 00:59:24.933300 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.933214 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vgb4g_68f27071-8614-4c45-8f17-ce4587ecd3d2/network-metrics-daemon/0.log" Jul 10 00:59:24.950537 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:24.950482 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vgb4g_68f27071-8614-4c45-8f17-ce4587ecd3d2/kube-rbac-proxy/0.log" Jul 10 00:59:25.802058 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.802026 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-controller/0.log" Jul 10 00:59:25.815813 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.815783 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/1.log" Jul 10 00:59:25.815813 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.815799 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovn-acl-logging/0.log" Jul 10 00:59:25.837238 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.837207 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/kube-rbac-proxy-node/0.log" Jul 10 00:59:25.858907 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.858877 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/kube-rbac-proxy-ovn-metrics/0.log" Jul 10 00:59:25.873175 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.873139 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/northd/0.log" Jul 10 00:59:25.890952 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.890904 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/nbdb/0.log" Jul 10 00:59:25.909754 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.909711 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/sbdb/0.log" Jul 10 00:59:25.994629 ip-10-0-137-8 kubenswrapper[2548]: I0710 00:59:25.994594 2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5lt5h_3732cd5e-42d6-4517-96a7-a6132fd22340/ovnkube-controller/0.log"