Jul 14 19:47:28.934979 ip-10-0-129-28 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:47:29.276368 ip-10-0-129-28 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:47:29.276368 ip-10-0-129-28 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:47:29.276368 ip-10-0-129-28 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:47:29.276368 ip-10-0-129-28 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:47:29.276368 ip-10-0-129-28 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:47:29.277842 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.277734 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:47:29.282940 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282926 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.282940 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282940 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282944 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282948 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282951 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282954 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282957 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282959 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282961 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282964 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282966 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282969 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282971 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282974 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282977 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282979 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282982 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282984 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282987 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282989 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.282992 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.283006 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283003 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283006 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283009 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283012 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283014 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283017 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283020 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283022 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283025 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283027 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283030 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283032 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283035 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283037 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283040 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283042 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283045 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283047 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283050 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283052 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.283464 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283054 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283058 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283062 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283065 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283068 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283070 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283074 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283076 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283079 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283082 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283085 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283087 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283089 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283098 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283102 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283105 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283108 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283111 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283113 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283116 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.283956 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283118 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283121 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283123 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283126 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283128 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283131 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283133 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283136 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283138 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283140 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283143 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283145 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283147 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283150 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283152 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283155 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283157 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283160 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283165 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.284435 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283168 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283171 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283173 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283176 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283178 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283181 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283183 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283187 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283190 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283192 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283195 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283197 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283200 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283202 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283204 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283207 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283210 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283213 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283217 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.284891 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283912 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283918 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283921 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283924 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283927 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283930 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283932 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283935 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283938 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283941 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283943 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283946 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283948 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283951 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283953 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283956 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283959 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283961 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283966 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.285341 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283970 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283974 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283978 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283981 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283984 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283987 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283989 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283992 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283994 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283997 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.283999 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284002 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284004 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284007 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284009 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284012 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284014 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284017 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284019 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284023 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.285814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284026 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284030 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284035 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284038 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284042 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284045 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284048 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284050 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284053 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284056 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284058 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284061 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284063 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284066 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284068 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284071 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284074 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284076 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284080 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.286294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284082 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284085 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284087 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284089 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284092 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284094 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284097 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284099 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284101 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284104 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284106 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284109 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284111 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284115 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284118 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284120 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284123 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284127 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284129 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284132 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.286769 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284134 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284137 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284140 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284142 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284144 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284147 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284150 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284152 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284154 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284157 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284160 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284162 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284165 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284167 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284170 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284173 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284175 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284177 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284180 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284182 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.287249 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284185 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284532 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284536 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284539 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284542 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284545 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284547 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284550 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284553 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284555 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284558 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284561 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284563 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284566 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284568 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284571 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284574 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284576 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284579 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284581 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.287744 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284583 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284586 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284589 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284591 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284594 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284596 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284600 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284604 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284606 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284609 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284612 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284614 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284617 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284620 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284623 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284625 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284628 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284631 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284634 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.288223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284636 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284639 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284641 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284643 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284646 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284648 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284651 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284653 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284656 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284659 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284662 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284664 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284667 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284669 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284672 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284694 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284697 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284701 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284705 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.288708 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284708 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284710 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284714 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284717 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284720 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284722 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284725 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284728 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284731 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284733 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284736 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284739 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284741 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284744 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284747 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284749 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284752 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284754 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284757 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284760 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.289162 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284763 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284765 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284770 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284773 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284776 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284778 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284781 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284783 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284786 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284789 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284791 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284793 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284796 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284799 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284801 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284803 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284806 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284808 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284811 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.289669 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284813 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284816 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.284818 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284881 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284888 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284894 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284898 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284902 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284905 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284910 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284914 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284917 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284920 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284923 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284926 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284934 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284937 2574 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284940 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284943 2574 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284946 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284949 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284953 2574 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284955 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284959 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:47:29.290140 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284962 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284965 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284968 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284971 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284974 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284977 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284980 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284983 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284985 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284988 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284991 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284995 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.284998 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285001 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285004 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285007 2574 flags.go:64] FLAG: --enable-server="true" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285010 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285015 2574 flags.go:64] FLAG: --event-burst="100" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285018 2574 flags.go:64] FLAG: --event-qps="50" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285021 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285024 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285026 2574 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285030 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285034 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285037 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:47:29.290720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285040 2574 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285043 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285046 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285048 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285051 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285054 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285057 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285060 2574 flags.go:64] FLAG: --feature-gates="" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285064 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285067 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285069 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285073 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285077 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285080 2574 flags.go:64] FLAG: --help="false" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285083 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285086 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285089 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285092 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285095 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285098 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285101 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285104 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285106 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:47:29.291618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285109 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285113 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285116 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285118 2574 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285121 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285124 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285127 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285130 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285133 2574 flags.go:64] FLAG: --lock-file="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285136 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285139 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285142 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285147 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285150 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285152 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285155 2574 flags.go:64] FLAG: --logging-format="text" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285158 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285161 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285164 2574 flags.go:64] FLAG: --manifest-url="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285166 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285170 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285175 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285179 2574 flags.go:64] FLAG: --max-pods="110" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285181 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285184 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:47:29.292479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285187 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285190 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285193 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285196 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285198 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285205 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285208 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285210 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285213 2574 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285216 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285221 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285224 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285227 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285229 2574 flags.go:64] FLAG: --port="10250" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285233 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285236 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-094b2a4e4da7afc67" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285239 2574 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285242 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285245 2574 flags.go:64] FLAG: --register-node="true" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285248 2574 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285251 2574 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285254 2574 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285257 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285259 2574 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285263 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:47:29.293109 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285266 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285269 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285272 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285274 2574 flags.go:64] FLAG: --runonce="false" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285278 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285281 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285284 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285287 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285290 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285292 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285295 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285298 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285300 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285303 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285306 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285308 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285312 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285314 2574 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285317 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285322 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285325 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285329 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285333 2574 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285335 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285338 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:47:29.293716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285340 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285343 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285346 2574 flags.go:64] FLAG: --v="2" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285350 2574 flags.go:64] FLAG: --version="false" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285354 2574 flags.go:64] FLAG: --vmodule="" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285357 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285360 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285469 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285473 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285475 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285478 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285482 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285485 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285487 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285490 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285492 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285495 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285497 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285500 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285502 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285505 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.294314 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285507 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285510 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285513 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285515 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285518 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285520 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285523 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285526 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285528 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285531 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285533 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285535 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285538 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285540 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285543 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285545 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285548 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285550 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285553 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285555 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.294831 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285558 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285560 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285564 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285566 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285569 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285571 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285574 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285576 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285579 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285581 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285584 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285586 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285588 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285593 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285596 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285599 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285602 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285605 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285607 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285611 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.295359 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285613 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285616 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285619 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285622 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285625 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285628 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285630 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285633 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285637 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285640 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285643 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285645 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285648 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285651 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285654 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285657 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285659 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285662 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285664 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.295854 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285667 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285669 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285671 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285674 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285690 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285693 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285695 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285698 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285701 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285705 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285707 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285710 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285713 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285715 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285718 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285720 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285723 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285725 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285727 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285730 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.296321 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285732 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285735 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285737 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285739 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285742 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.285744 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.285750 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.293860 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.293877 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.294859 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.294867 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294924 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294929 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294932 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294935 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.296824 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294937 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294940 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294943 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294945 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294948 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294951 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294954 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294957 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294960 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294962 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294965 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294967 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294970 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294973 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294975 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294978 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294980 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294984 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294986 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294989 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294991 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.297206 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294994 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294996 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.294999 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295001 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295004 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295007 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295010 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295012 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295015 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295019 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295023 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295026 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295028 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295031 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295034 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295036 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295038 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295041 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295043 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.297721 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295046 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295049 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295052 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295055 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295057 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295060 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295062 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295065 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295067 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295071 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295073 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295076 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295078 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295081 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295083 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295086 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295089 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295092 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295094 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295097 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.298169 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295099 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295102 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295104 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295107 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295109 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295113 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295117 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295120 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295123 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295126 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295128 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295131 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295134 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295136 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295140 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295144 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295147 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295149 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295152 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.298734 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295154 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295157 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295160 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295162 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295165 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295168 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295170 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295173 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295175 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295177 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295180 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295183 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295185 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295187 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295190 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.299226 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295192 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.295197 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295312 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295317 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295320 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295323 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295326 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295329 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295331 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295334 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295336 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295339 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295342 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295344 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295350 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:47:29.299585 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295353 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295357 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295361 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295364 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295367 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295370 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295373 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295375 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295378 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295382 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295386 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295389 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295391 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295394 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295396 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295399 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295401 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295404 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295407 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:47:29.299966 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295409 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295412 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295414 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295417 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295419 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295422 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295424 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295427 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295429 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295432 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295434 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295436 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295439 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295442 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295444 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295447 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295450 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295452 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295455 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295457 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:47:29.300431 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295460 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295479 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295482 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295485 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295488 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295491 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295494 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295497 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295500 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295502 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295505 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295507 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295510 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295513 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295515 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295518 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295520 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295523 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295525 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295528 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:47:29.300942 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295530 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295533 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295535 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295538 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295540 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295542 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295545 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295548 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295551 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295553 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295557 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295559 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295562 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295564 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295567 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295569 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295572 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295574 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295577 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:47:29.301419 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295579 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295582 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295584 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295586 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295589 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295592 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295596 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:29.295598 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.295603 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.295722 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.298291 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.299523 2574 server.go:1019] "Starting client certificate rotation" Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.299628 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:47:29.301875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.299667 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:47:29.302306 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.302293 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:47:29.304805 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.304790 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:47:29.314045 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.314029 2574 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:47:29.316766 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.316752 2574 log.go:25] "Validated CRI v1 image API" Jul 14 19:47:29.318078 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.318062 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:47:29.320707 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.320670 2574 fs.go:135] Filesystem UUIDs: map[4d61bcc1-be20-477c-ad53-1be6852f90bb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ede22053-9ae4-4f75-b503-583d357fe380:/dev/nvme0n1p4] Jul 14 19:47:29.320778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.320707 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:47:29.324288 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.324266 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:47:29.326453 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.326346 2574 manager.go:216] Machine: {Timestamp:2026-07-14 19:47:29.325224867 +0000 UTC m=+0.287372058 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104170 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec275b62f5ab230918de4972010a21bc SystemUUID:ec275b62-f5ab-2309-18de-4972010a21bc BootID:724e7caa-b773-4417-9453-e6d41756d536 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:a4:e4:2c:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:a4:e4:2c:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:5a:59:34:87:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:47:29.326453 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.326448 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:47:29.326566 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.326516 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:47:29.328034 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.328006 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:47:29.328160 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.328038 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:47:29.329617 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.329606 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:47:29.329657 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.329619 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:47:29.329657 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.329626 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:47:29.329657 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.329639 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:47:29.330470 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.330461 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:47:29.334234 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334223 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:47:29.334341 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334332 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:47:29.334372 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334362 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:47:29.334401 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334372 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:47:29.334401 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334385 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:47:29.334401 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334393 2574 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:47:29.334503 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334402 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:47:29.334759 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334747 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:47:29.334814 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.334771 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:47:29.336819 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.336799 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:47:29.338560 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.338543 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:47:29.338659 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.338582 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:47:29.339235 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339222 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339242 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339251 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339259 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339272 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339280 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339292 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339301 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:47:29.339312 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339311 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:47:29.339550 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339320 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:47:29.339550 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339332 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:47:29.339550 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339341 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:47:29.339853 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339842 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:47:29.339910 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.339856 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:47:29.342925 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.342909 2574 server.go:1268] "Started kubelet" Jul 14 19:47:29.343164 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.343056 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:47:29.343648 ip-10-0-129-28 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:47:29.343778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.343693 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:47:29.343778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.343770 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:47:29.344616 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.344443 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:47:29.345147 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.345118 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:47:29.345232 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.345119 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:47:29.345542 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.345517 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:47:29.345902 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.345890 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:47:29.347909 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.347891 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:47:29.349097 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.349078 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vqr5" Jul 14 19:47:29.350604 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.349861 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-28.ec2.internal.18c23fb5139de1ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-28.ec2.internal,UID:ip-10-0-129-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-28.ec2.internal,},FirstTimestamp:2026-07-14 19:47:29.34288225 +0000 UTC m=+0.305029440,LastTimestamp:2026-07-14 19:47:29.34288225 +0000 UTC m=+0.305029440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-28.ec2.internal,}" Jul 14 19:47:29.351107 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.351092 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:47:29.351154 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.351107 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:47:29.351233 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.351213 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.352279 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352091 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:47:29.352279 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352118 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:47:29.352279 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.351979 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:47:29.352462 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352314 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:47:29.352462 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352325 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:47:29.352790 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352767 2574 factory.go:153] Registering CRI-O factory Jul 14 19:47:29.352790 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352787 2574 factory.go:223] Registration of the crio container factory successfully Jul 14 19:47:29.352917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352836 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:47:29.352917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352844 2574 factory.go:55] Registering systemd factory Jul 14 19:47:29.352917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352853 2574 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:47:29.352917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352892 2574 factory.go:103] Registering Raw factory Jul 14 19:47:29.352917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.352907 2574 manager.go:1196] Started watching for new ooms in manager Jul 14 19:47:29.353295 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.353276 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 19:47:29.353382 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.353369 2574 manager.go:349] Starting recovery of all containers Jul 14 19:47:29.353382 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.353374 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 19:47:29.359946 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.359922 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vqr5" Jul 14 19:47:29.365268 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.365242 2574 manager.go:354] Recovery completed Jul 14 19:47:29.369518 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.369504 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.371280 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371266 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.371346 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371293 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.371346 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371303 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.371909 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371895 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:47:29.371947 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371910 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:47:29.371947 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371920 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.371947 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.371925 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:47:29.372611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.372592 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.372697 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.372630 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.372697 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.372640 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.373046 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.372982 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-28.ec2.internal.18c23fb5154f3454 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-28.ec2.internal,UID:ip-10-0-129-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-28.ec2.internal,},FirstTimestamp:2026-07-14 19:47:29.371280468 +0000 UTC m=+0.333427657,LastTimestamp:2026-07-14 19:47:29.371280468 +0000 UTC m=+0.333427657,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-28.ec2.internal,}" Jul 14 19:47:29.374334 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.374320 2574 policy_none.go:49] "None policy: Start" Jul 14 19:47:29.374384 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.374341 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:47:29.374384 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.374350 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:47:29.376374 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.376279 2574 policy_none.go:47] "Start" Jul 14 19:47:29.412214 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412199 2574 manager.go:337] "Starting Device Plugin manager" Jul 14 19:47:29.412300 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.412232 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:47:29.412300 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412244 2574 server.go:87] "Starting device plugin registration server" Jul 14 19:47:29.412512 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412499 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:47:29.412590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412523 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:47:29.412638 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412605 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:47:29.412735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412708 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:47:29.412735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.412721 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:47:29.413224 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.413195 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:47:29.413301 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.413267 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.474027 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.474001 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:47:29.476267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.475421 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:47:29.476267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.475456 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:47:29.476267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.475474 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:47:29.476267 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.475508 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:47:29.478309 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.478294 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:29.513587 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.513559 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.514407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.514392 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.514461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.514427 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.514461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.514447 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.514520 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.514469 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.523456 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.523443 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.523520 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.523463 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-28.ec2.internal\": node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.540024 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.539987 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.576199 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.576172 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal"] Jul 14 19:47:29.576243 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.576236 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.577557 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.577537 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.577609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.577567 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.577609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.577577 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.579926 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.579914 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.580079 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.580066 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.580112 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.580096 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.580582 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.580556 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.580634 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.580602 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.580634 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.580621 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.581089 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.581074 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.581154 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.581101 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.581154 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.581116 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.583249 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.583234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.583307 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.583260 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:47:29.583991 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.583974 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:47:29.584058 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.584004 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:47:29.584058 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.584017 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:47:29.616186 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.616169 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-28.ec2.internal\" not found" node="ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.620386 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.620370 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-28.ec2.internal\" not found" node="ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.640185 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.640166 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.653465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.653439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.653537 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.653475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.653537 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.653499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43ba1ebae5d7ee21a4751f58c11fde6b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-28.ec2.internal\" (UID: \"43ba1ebae5d7ee21a4751f58c11fde6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.740981 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.740961 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.754325 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.754376 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.754376 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43ba1ebae5d7ee21a4751f58c11fde6b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-28.ec2.internal\" (UID: \"43ba1ebae5d7ee21a4751f58c11fde6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.754446 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43ba1ebae5d7ee21a4751f58c11fde6b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-28.ec2.internal\" (UID: \"43ba1ebae5d7ee21a4751f58c11fde6b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.754446 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.754446 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.754418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b31dba48238fdc286bfab6bd4db2f24f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal\" (UID: \"b31dba48238fdc286bfab6bd4db2f24f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.841798 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.841722 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:29.918274 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.918242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.923593 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:29.923256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:29.942152 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:29.942124 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:30.042696 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.042658 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:30.143240 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.143180 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:30.243874 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.243850 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-28.ec2.internal\" not found" Jul 14 19:47:30.244397 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.244383 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:30.251424 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.251405 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" Jul 14 19:47:30.263871 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.263851 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:47:30.265618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.265605 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" Jul 14 19:47:30.272023 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.272000 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:47:30.299136 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.299112 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:47:30.299484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.299227 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:47:30.299484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.299278 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:47:30.335437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.335416 2574 apiserver.go:52] "Watching apiserver" Jul 14 19:47:30.351794 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.351771 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:47:30.353474 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.353456 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:47:30.353834 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.353815 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8rvfr","kube-system/konnectivity-agent-cpz46","kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc","openshift-cluster-node-tuning-operator/tuned-dzpk9","openshift-dns/node-resolver-dltdc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal","openshift-multus/multus-additional-cni-plugins-tpn29","openshift-image-registry/node-ca-vw69b","openshift-multus/multus-6ttml","openshift-multus/network-metrics-daemon-s5c8w","openshift-network-diagnostics/network-check-target-zvd8p","openshift-network-operator/iptables-alerter-87hfz","openshift-ovn-kubernetes/ovnkube-node-68t8w"] Jul 14 19:47:30.356282 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.356266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.358384 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.358368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.358882 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.358867 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.358959 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.358944 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.359022 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.359004 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dm5q6\"" Jul 14 19:47:30.359398 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.359386 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:47:30.359453 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.359387 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:47:30.360588 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.360567 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.360740 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.360725 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:47:30.360802 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.360768 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.361569 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.361546 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmfjd\"" Jul 14 19:47:30.361631 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.361550 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.362916 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.362903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.363414 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.363398 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:47:30.363480 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.363451 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:47:30.363480 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.363459 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:47:30.363550 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.363529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.364534 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.364521 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cl9st\"" Jul 14 19:47:30.364615 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.364601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:47:30.364652 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.364613 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.365135 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.365119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.365361 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.365345 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:47:30.365597 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.365583 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.366522 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.366492 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:42:30 +0000 UTC" deadline="2028-02-03 03:02:34.497696124 +0000 UTC" Jul 14 19:47:30.366522 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.366520 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13639h15m4.131179243s" Jul 14 19:47:30.366728 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.366597 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.366728 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.366603 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tlgjd\"" Jul 14 19:47:30.367258 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.367243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4cnbg\"" Jul 14 19:47:30.367366 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.367343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.367710 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.367694 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.367781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.367695 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.369653 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.369636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.370370 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.370352 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.370664 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.370645 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bg2g8\"" Jul 14 19:47:30.370756 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.370718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.372705 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.372665 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:47:30.372914 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.372896 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhmkp\"" Jul 14 19:47:30.373297 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.373286 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:47:30.373538 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.373520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.373764 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.373529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:47:30.375764 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.375749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:47:30.375931 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.375905 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:47:30.376000 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.375939 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.376000 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.375973 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:47:30.376463 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.376448 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sbmtw\"" Jul 14 19:47:30.378375 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.378357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:47:30.378455 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.378388 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7svnd\"" Jul 14 19:47:30.378455 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.378394 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:47:30.378717 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.378705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.378782 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.378764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:30.380929 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.380913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.380988 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.380973 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:30.383075 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.383062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:30.383137 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.383123 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:30.386822 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.386806 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cs778" Jul 14 19:47:30.397225 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.397139 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cs778" Jul 14 19:47:30.453372 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.453355 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:47:30.457906 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-slash\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.457962 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-system-cni-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.457962 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.457962 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-node-log\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458065 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-env-overrides\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458065 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.457990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovn-node-metrics-cert\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458065 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvk2z\" (UniqueName: \"kubernetes.io/projected/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-kube-api-access-vvk2z\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458065 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw455\" (UniqueName: \"kubernetes.io/projected/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-kube-api-access-pw455\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-cnibin\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-etc-kubernetes\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-tmp\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.458203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458199 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-systemd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-socket-dir-parent\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-multus\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-multus-certs\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-etc-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-bin\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-netd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-modprobe-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-device-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-conf\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1881a13-f947-4dbd-b8cf-f89d4a860b61-agent-certs\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cni-binary-copy\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-conf-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysconfig\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-var-lib-kubelet\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-etc-tuned\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-iptables-alerter-script\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-os-release\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-log-socket\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-sys-fs\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458693 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1881a13-f947-4dbd-b8cf-f89d4a860b61-konnectivity-ca\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-system-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-hostroot\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-ovn\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cnibin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-daemon-config\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.458893 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-kubelet\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-host\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-config\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-script-lib\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-etc-selinux\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvb7j\" (UniqueName: \"kubernetes.io/projected/974dec77-4151-4c86-955b-61d7ca53da5d-kube-api-access-pvb7j\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z4s8\" (UniqueName: \"kubernetes.io/projected/80dfc55c-0e33-4820-b465-87fa724e7757-kube-api-access-2z4s8\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458926 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-host\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-serviceca\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdk8\" (UniqueName: \"kubernetes.io/projected/99b20e35-bc5e-44f8-94de-2828be472ec0-kube-api-access-bcdk8\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.458986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-kubelet\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-socket-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-systemd\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-kubelet-config\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.459523 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-dbus\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-bin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-var-lib-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-registration-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nsz\" (UniqueName: \"kubernetes.io/projected/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kube-api-access-52nsz\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-host-slash\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867zq\" (UniqueName: \"kubernetes.io/projected/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-kube-api-access-867zq\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-hosts-file\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-netns\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-lib-modules\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjltf\" (UniqueName: \"kubernetes.io/projected/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-kube-api-access-qjltf\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-k8s-cni-cncf-io\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-systemd-units\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-netns\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.460085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-kubernetes\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.460559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-run\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.460559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459667 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-sys\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.460559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-os-release\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.460559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-tmp-dir\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.460559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.459792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftk5p\" (UniqueName: \"kubernetes.io/projected/19dc80ba-52eb-4761-918f-e5b4be8ee2df-kube-api-access-ftk5p\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.560487 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-system-cni-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.560487 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-node-log\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-env-overrides\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovn-node-metrics-cert\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvk2z\" (UniqueName: \"kubernetes.io/projected/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-kube-api-access-vvk2z\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-system-cni-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.560611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-node-log\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw455\" (UniqueName: \"kubernetes.io/projected/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-kube-api-access-pw455\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-cnibin\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-etc-kubernetes\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-tmp\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-systemd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-socket-dir-parent\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-multus\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-multus-certs\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-etc-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-bin\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-cnibin\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-netd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560981 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-multus-certs\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-modprobe-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-systemd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-device-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-etc-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561104 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-device-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.560984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-multus\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-socket-dir-parent\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-bin\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-env-overrides\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561162 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-cni-netd\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-conf\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1881a13-f947-4dbd-b8cf-f89d4a860b61-agent-certs\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-etc-kubernetes\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.561781 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-modprobe-d\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cni-binary-copy\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-conf-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561311 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-conf-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysctl-conf\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysconfig\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-var-lib-kubelet\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-var-lib-kubelet\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-etc-tuned\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561500 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-sysconfig\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-iptables-alerter-script\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-os-release\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-log-socket\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.562609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-os-release\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-sys-fs\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-log-socket\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1881a13-f947-4dbd-b8cf-f89d4a860b61-konnectivity-ca\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-system-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-hostroot\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-sys-fs\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-hostroot\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-ovn\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-system-cni-dir\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-run-ovn\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kubelet-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cnibin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-daemon-config\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.561995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-kubelet\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.563434 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-kubelet\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-iptables-alerter-script\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-host\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562064 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cni-binary-copy\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-host\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562040 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-cnibin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-config\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-script-lib\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-etc-selinux\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvb7j\" (UniqueName: \"kubernetes.io/projected/974dec77-4151-4c86-955b-61d7ca53da5d-kube-api-access-pvb7j\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.562321 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z4s8\" (UniqueName: \"kubernetes.io/projected/80dfc55c-0e33-4820-b465-87fa724e7757-kube-api-access-2z4s8\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.562377 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.062359347 +0000 UTC m=+2.024506544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-host\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-serviceca\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.564179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562455 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/19dc80ba-52eb-4761-918f-e5b4be8ee2df-multus-daemon-config\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdk8\" (UniqueName: \"kubernetes.io/projected/99b20e35-bc5e-44f8-94de-2828be472ec0-kube-api-access-bcdk8\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-kubelet\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-socket-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-systemd\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-kubelet-config\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-dbus\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562666 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-dbus\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-kubelet\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-socket-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-systemd\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daa50ea0-e19e-4c48-9afc-adc079abf2d5-kubelet-config\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.562961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-host\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-etc-selinux\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-config\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563182 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-serviceca\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.564818 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-bin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-var-lib-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-registration-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nsz\" (UniqueName: \"kubernetes.io/projected/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kube-api-access-52nsz\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80dfc55c-0e33-4820-b465-87fa724e7757-cni-binary-copy\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563346 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-host-slash\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867zq\" (UniqueName: \"kubernetes.io/projected/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-kube-api-access-867zq\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-hosts-file\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-netns\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-lib-modules\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-var-lib-cni-bin\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjltf\" (UniqueName: \"kubernetes.io/projected/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-kube-api-access-qjltf\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-var-lib-openvswitch\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-k8s-cni-cncf-io\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovnkube-script-lib\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-systemd-units\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.565466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-k8s-cni-cncf-io\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9a78e5d3-fc7d-4702-bb80-9a78631e5689-registration-dir\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19dc80ba-52eb-4761-918f-e5b4be8ee2df-host-run-netns\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-hosts-file\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563587 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-systemd-units\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-netns\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-kubernetes\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-run\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563668 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-lib-modules\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-sys\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-os-release\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-etc-kubernetes\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-tmp-dir\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-run-netns\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftk5p\" (UniqueName: \"kubernetes.io/projected/19dc80ba-52eb-4761-918f-e5b4be8ee2df-kube-api-access-ftk5p\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.563786 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-host-slash\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:30.566094 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.563844 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.063828434 +0000 UTC m=+2.025975614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-run\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-slash\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80dfc55c-0e33-4820-b465-87fa724e7757-os-release\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-host-slash\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.563968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/974dec77-4151-4c86-955b-61d7ca53da5d-sys\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.564225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-tmp-dir\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.564365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1881a13-f947-4dbd-b8cf-f89d4a860b61-konnectivity-ca\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.565125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1881a13-f947-4dbd-b8cf-f89d4a860b61-agent-certs\") pod \"konnectivity-agent-cpz46\" (UID: \"a1881a13-f947-4dbd-b8cf-f89d4a860b61\") " pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.565261 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-tmp\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.565291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/974dec77-4151-4c86-955b-61d7ca53da5d-etc-tuned\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.566584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.565767 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-ovn-node-metrics-cert\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.570949 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.570910 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:30.570949 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.570933 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:30.570949 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.570947 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:30.571096 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:30.571009 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:31.070993068 +0000 UTC m=+2.033140263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:30.571574 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.571530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvk2z\" (UniqueName: \"kubernetes.io/projected/3c642359-2a0f-438f-84eb-e1a59e6d2fb5-kube-api-access-vvk2z\") pod \"ovnkube-node-68t8w\" (UID: \"3c642359-2a0f-438f-84eb-e1a59e6d2fb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:30.573252 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.573226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftk5p\" (UniqueName: \"kubernetes.io/projected/19dc80ba-52eb-4761-918f-e5b4be8ee2df-kube-api-access-ftk5p\") pod \"multus-6ttml\" (UID: \"19dc80ba-52eb-4761-918f-e5b4be8ee2df\") " pod="openshift-multus/multus-6ttml" Jul 14 19:47:30.573788 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.573764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z4s8\" (UniqueName: \"kubernetes.io/projected/80dfc55c-0e33-4820-b465-87fa724e7757-kube-api-access-2z4s8\") pod \"multus-additional-cni-plugins-tpn29\" (UID: \"80dfc55c-0e33-4820-b465-87fa724e7757\") " pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:30.574368 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.574140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nsz\" (UniqueName: \"kubernetes.io/projected/9a78e5d3-fc7d-4702-bb80-9a78631e5689-kube-api-access-52nsz\") pod \"aws-ebs-csi-driver-node-567nc\" (UID: \"9a78e5d3-fc7d-4702-bb80-9a78631e5689\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:30.574368 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.574174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvb7j\" (UniqueName: \"kubernetes.io/projected/974dec77-4151-4c86-955b-61d7ca53da5d-kube-api-access-pvb7j\") pod \"tuned-dzpk9\" (UID: \"974dec77-4151-4c86-955b-61d7ca53da5d\") " pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:30.574368 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.574313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjltf\" (UniqueName: \"kubernetes.io/projected/4e6499e8-d01c-4ff3-b126-6ee1237e17c4-kube-api-access-qjltf\") pod \"node-resolver-dltdc\" (UID: \"4e6499e8-d01c-4ff3-b126-6ee1237e17c4\") " pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:30.574920 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.574903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdk8\" (UniqueName: \"kubernetes.io/projected/99b20e35-bc5e-44f8-94de-2828be472ec0-kube-api-access-bcdk8\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:30.575048 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.575034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw455\" (UniqueName: \"kubernetes.io/projected/ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d-kube-api-access-pw455\") pod \"node-ca-vw69b\" (UID: \"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d\") " pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:30.575799 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:30.575781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867zq\" (UniqueName: \"kubernetes.io/projected/09afce7d-82ab-41b5-8c8d-8dbddc9a619e-kube-api-access-867zq\") pod \"iptables-alerter-87hfz\" (UID: \"09afce7d-82ab-41b5-8c8d-8dbddc9a619e\") " pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:31.806889 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.806838 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:31.814881 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.814862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6ttml" Jul 14 19:47:31.832743 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.832719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-87hfz" Jul 14 19:47:31.846221 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.846202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:31.850961 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.850945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" Jul 14 19:47:31.865387 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.865370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" Jul 14 19:47:31.869158 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.869141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dltdc" Jul 14 19:47:31.882625 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.882608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vw69b" Jul 14 19:47:31.901082 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.901063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tpn29" Jul 14 19:47:31.905735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.905719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:31.910934 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:31.910903 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31dba48238fdc286bfab6bd4db2f24f.slice/crio-d7ad4dca82d3de78e86d15bb1e4f8a430ff41ec43fc5c77ab752f357f1bd3a39 WatchSource:0}: Error finding container d7ad4dca82d3de78e86d15bb1e4f8a430ff41ec43fc5c77ab752f357f1bd3a39: Status 404 returned error can't find the container with id d7ad4dca82d3de78e86d15bb1e4f8a430ff41ec43fc5c77ab752f357f1bd3a39 Jul 14 19:47:31.911473 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:31.911454 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecffb1fb_7b0d_4884_951f_24ce4fc3ae1d.slice/crio-654052663fe5eb0741b990a4094b6ee0e3aa299df007019ad574c04acba29007 WatchSource:0}: Error finding container 654052663fe5eb0741b990a4094b6ee0e3aa299df007019ad574c04acba29007: Status 404 returned error can't find the container with id 654052663fe5eb0741b990a4094b6ee0e3aa299df007019ad574c04acba29007 Jul 14 19:47:31.912644 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:31.912621 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dfc55c_0e33_4820_b465_87fa724e7757.slice/crio-3b6adcdba6cfdc2e318ba326d5007f4ccc23cbc350715cc3f3cab3cb4518154f WatchSource:0}: Error finding container 3b6adcdba6cfdc2e318ba326d5007f4ccc23cbc350715cc3f3cab3cb4518154f: Status 404 returned error can't find the container with id 3b6adcdba6cfdc2e318ba326d5007f4ccc23cbc350715cc3f3cab3cb4518154f Jul 14 19:47:31.915860 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:31.914407 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ba1ebae5d7ee21a4751f58c11fde6b.slice/crio-b251e9c810e6ac47318b8812659ec3dfb6efff4b537cdd0b462d40538ad0f0db WatchSource:0}: Error finding container b251e9c810e6ac47318b8812659ec3dfb6efff4b537cdd0b462d40538ad0f0db: Status 404 returned error can't find the container with id b251e9c810e6ac47318b8812659ec3dfb6efff4b537cdd0b462d40538ad0f0db Jul 14 19:47:31.918197 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:31.918166 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:47:31.919223 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:31.919101 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1881a13_f947_4dbd_b8cf_f89d4a860b61.slice/crio-32cd83c20de84a6d7a91acc24d4123f8dcaa98a7e643b86d500dfb92bd2ae19e WatchSource:0}: Error finding container 32cd83c20de84a6d7a91acc24d4123f8dcaa98a7e643b86d500dfb92bd2ae19e: Status 404 returned error can't find the container with id 32cd83c20de84a6d7a91acc24d4123f8dcaa98a7e643b86d500dfb92bd2ae19e Jul 14 19:47:32.035170 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.035141 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:32.120589 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:32.120568 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dc80ba_52eb_4761_918f_e5b4be8ee2df.slice/crio-d0132854b8c73f4f08bcb9bfd3f1f7f78ac0029d7b4168066469a7644f329b38 WatchSource:0}: Error finding container d0132854b8c73f4f08bcb9bfd3f1f7f78ac0029d7b4168066469a7644f329b38: Status 404 returned error can't find the container with id d0132854b8c73f4f08bcb9bfd3f1f7f78ac0029d7b4168066469a7644f329b38 Jul 14 19:47:32.185595 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.185564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:32.185714 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.185613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:32.185749 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.185712 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:32.185749 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.185713 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:32.185808 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.185770 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.185753952 +0000 UTC m=+3.030090820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:32.185808 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.185794 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.185784739 +0000 UTC m=+3.030121591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:32.287048 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.287004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:32.287297 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.287247 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:32.287297 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.287300 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:32.287478 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.287315 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:32.287478 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:32.287376 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:33.287356378 +0000 UTC m=+3.131693247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:32.314169 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.314136 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:32.516122 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:32.516090 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09afce7d_82ab_41b5_8c8d_8dbddc9a619e.slice/crio-b07a27d096173df788af8cb69a3064b08d10ebcca19b33b418d68ffd8e01ff0b WatchSource:0}: Error finding container b07a27d096173df788af8cb69a3064b08d10ebcca19b33b418d68ffd8e01ff0b: Status 404 returned error can't find the container with id b07a27d096173df788af8cb69a3064b08d10ebcca19b33b418d68ffd8e01ff0b Jul 14 19:47:32.517754 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.517704 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:42:31 +0000 UTC" deadline="2028-02-12 00:32:08.697289844 +0000 UTC" Jul 14 19:47:32.517856 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.517755 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13852h44m36.179537726s" Jul 14 19:47:32.604847 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.604785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" event={"ID":"43ba1ebae5d7ee21a4751f58c11fde6b","Type":"ContainerStarted","Data":"b251e9c810e6ac47318b8812659ec3dfb6efff4b537cdd0b462d40538ad0f0db"} Jul 14 19:47:32.614054 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.613971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vw69b" event={"ID":"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d","Type":"ContainerStarted","Data":"654052663fe5eb0741b990a4094b6ee0e3aa299df007019ad574c04acba29007"} Jul 14 19:47:32.617462 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.617337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-87hfz" event={"ID":"09afce7d-82ab-41b5-8c8d-8dbddc9a619e","Type":"ContainerStarted","Data":"b07a27d096173df788af8cb69a3064b08d10ebcca19b33b418d68ffd8e01ff0b"} Jul 14 19:47:32.620477 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.620440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" event={"ID":"974dec77-4151-4c86-955b-61d7ca53da5d","Type":"ContainerStarted","Data":"62de79069885e621a851bf1a0cc0653c0a18fd92e849e95cd3e75e111da75442"} Jul 14 19:47:32.625435 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.625410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6ttml" event={"ID":"19dc80ba-52eb-4761-918f-e5b4be8ee2df","Type":"ContainerStarted","Data":"d0132854b8c73f4f08bcb9bfd3f1f7f78ac0029d7b4168066469a7644f329b38"} Jul 14 19:47:32.629128 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.629095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cpz46" event={"ID":"a1881a13-f947-4dbd-b8cf-f89d4a860b61","Type":"ContainerStarted","Data":"32cd83c20de84a6d7a91acc24d4123f8dcaa98a7e643b86d500dfb92bd2ae19e"} Jul 14 19:47:32.631952 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.631914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerStarted","Data":"3b6adcdba6cfdc2e318ba326d5007f4ccc23cbc350715cc3f3cab3cb4518154f"} Jul 14 19:47:32.634217 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:32.634182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" event={"ID":"b31dba48238fdc286bfab6bd4db2f24f","Type":"ContainerStarted","Data":"d7ad4dca82d3de78e86d15bb1e4f8a430ff41ec43fc5c77ab752f357f1bd3a39"} Jul 14 19:47:32.694543 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:32.694511 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6499e8_d01c_4ff3_b126_6ee1237e17c4.slice/crio-1f08f522e4fd4049489811d846586ebbf47f4672bba16808b69e99d3ca083233 WatchSource:0}: Error finding container 1f08f522e4fd4049489811d846586ebbf47f4672bba16808b69e99d3ca083233: Status 404 returned error can't find the container with id 1f08f522e4fd4049489811d846586ebbf47f4672bba16808b69e99d3ca083233 Jul 14 19:47:32.709256 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:47:32.709231 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c642359_2a0f_438f_84eb_e1a59e6d2fb5.slice/crio-94c5dc8b6e41bd850ca17ac4e3a06ff9748399acb67d9301fb26b52669a5e9f7 WatchSource:0}: Error finding container 94c5dc8b6e41bd850ca17ac4e3a06ff9748399acb67d9301fb26b52669a5e9f7: Status 404 returned error can't find the container with id 94c5dc8b6e41bd850ca17ac4e3a06ff9748399acb67d9301fb26b52669a5e9f7 Jul 14 19:47:33.184032 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.184000 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:47:33.193243 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.193218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:33.193373 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.193278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:33.193444 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.193403 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:33.193520 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.193462 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:35.193444819 +0000 UTC m=+5.037781676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:33.193850 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.193834 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:33.193934 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.193886 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:35.193871593 +0000 UTC m=+5.038208447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:33.293881 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.293834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:33.294052 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.294026 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:33.294052 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.294050 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:33.294161 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.294063 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:33.294161 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.294120 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:35.294103437 +0000 UTC m=+5.138440307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:33.518506 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.518398 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:42:31 +0000 UTC" deadline="2028-01-07 18:25:35.169468496 +0000 UTC" Jul 14 19:47:33.518506 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.518444 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13006h38m1.651029309s" Jul 14 19:47:33.594223 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.594189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:33.594384 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.594318 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:33.594773 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.594755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:33.594858 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.594840 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:33.594924 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.594891 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:33.594991 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:33.594973 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:33.641704 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.641610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"94c5dc8b6e41bd850ca17ac4e3a06ff9748399acb67d9301fb26b52669a5e9f7"} Jul 14 19:47:33.645414 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.645387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" event={"ID":"9a78e5d3-fc7d-4702-bb80-9a78631e5689","Type":"ContainerStarted","Data":"e98ca8112a488f6a4015ccf7ee9ecdd41d459dc98cfb44e0a2410c09533a1afe"} Jul 14 19:47:33.647409 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:33.647310 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dltdc" event={"ID":"4e6499e8-d01c-4ff3-b126-6ee1237e17c4","Type":"ContainerStarted","Data":"1f08f522e4fd4049489811d846586ebbf47f4672bba16808b69e99d3ca083233"} Jul 14 19:47:35.209420 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.209382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:35.209903 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.209469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:35.209903 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.209708 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:35.209903 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.209825 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.20979265 +0000 UTC m=+9.054129507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:35.210349 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.210329 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:35.210445 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.210389 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.210372154 +0000 UTC m=+9.054709008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:35.310029 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.309989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:35.310244 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.310227 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:35.310315 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.310254 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:35.310315 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.310267 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:35.310409 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.310342 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:39.310324524 +0000 UTC m=+9.154661380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:35.593797 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.593706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:35.593952 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.593841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:35.594184 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.594166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:35.594278 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.594246 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:35.594328 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:35.594290 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:35.594384 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:35.594339 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:37.594188 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:37.594149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:37.594685 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:37.594290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:37.594742 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:37.594705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:37.594818 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:37.594791 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:37.594972 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:37.594847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:37.594972 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:37.594918 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.241395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.241465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.241600 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.241658 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:47.241639148 +0000 UTC m=+17.085976005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.241710 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:39.241999 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.241767 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:47.241750053 +0000 UTC m=+17.086086914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:39.343088 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.342788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:39.343088 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.342970 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:39.343088 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.342991 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:39.343088 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.343004 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:39.343088 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.343055 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:47.34303957 +0000 UTC m=+17.187376436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:39.593620 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.593533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:39.593778 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.593713 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:39.594217 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.594193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:39.594405 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.594380 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:39.594480 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:39.594471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:39.594648 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:39.594625 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:41.594129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:41.594245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:41.594360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:41.594249 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:41.594253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:41.594573 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:41.594445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:43.594133 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:43.594100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:43.594529 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:43.594196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:43.594529 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:43.594205 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:43.594529 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:43.594230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:43.594529 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:43.594307 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:43.594529 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:43.594391 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:45.594178 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:45.593980 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:45.594905 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:45.594033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:45.594905 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:45.594267 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:45.594905 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:45.594045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:45.594905 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:45.594323 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:45.594905 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:45.594392 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:47.299849 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.299815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:47.300293 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.299882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:47.300293 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.299990 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:47.300293 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.299993 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:47.300293 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.300057 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:03.300037999 +0000 UTC m=+33.144374875 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:47:47.300293 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.300076 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:03.300066441 +0000 UTC m=+33.144403299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:47:47.400401 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.400372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:47.400580 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.400522 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:47:47.400580 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.400536 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:47:47.400580 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.400546 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:47.400682 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.400588 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:48:03.400576442 +0000 UTC m=+33.244913293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:47:47.594283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.594203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:47.594283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.594235 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:47.594283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:47.594239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:47.594548 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.594314 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:47.594548 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.594402 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:47.594548 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:47.594506 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:49.594439 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:49.594404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:49.594439 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:49.594454 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:49.594997 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:49.594404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:49.594997 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:49.594560 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:49.594997 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:49.594701 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:49.594997 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:49.594793 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:51.594118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:51.594227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:51.594241 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:51.594258 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:51.594340 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:51.594521 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:51.594446 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:52.679610 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.679407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"d6a914468419db51852f2ee135e3396b3882ba4acdddce25499df5f84d121bf3"} Jul 14 19:47:52.680130 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.679654 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"b2e9ecf182ba2320d5f0b6a3b34bd98a22a46a1917155fa21de82ece29a329fd"} Jul 14 19:47:52.680130 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.679699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"8c7f9b8d74cadc745267834873b1fc6b9048e956a5ad4a109eebc6a9bf714131"} Jul 14 19:47:52.680802 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.680777 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" event={"ID":"9a78e5d3-fc7d-4702-bb80-9a78631e5689","Type":"ContainerStarted","Data":"2f19d88f5370faf7c89fed85f729aff76e795286d89c080500d3e2282cb5a1ab"} Jul 14 19:47:52.682122 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.682101 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6ttml" event={"ID":"19dc80ba-52eb-4761-918f-e5b4be8ee2df","Type":"ContainerStarted","Data":"3efa0cd825f9228a651d1663a6c96bef2d1926283ef416b4423a1f73ecadf83e"} Jul 14 19:47:52.683479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.683459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vw69b" event={"ID":"ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d","Type":"ContainerStarted","Data":"e33ea8bac6872474d855f1f911682d7a87ef003d5863768519bbdbbb7fe29402"} Jul 14 19:47:52.684783 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.684757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dltdc" event={"ID":"4e6499e8-d01c-4ff3-b126-6ee1237e17c4","Type":"ContainerStarted","Data":"a8f4204cbab34980f70ba3f89f99a2de337cfdd793bbbd009b904d6838b97054"} Jul 14 19:47:52.686162 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.686136 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" event={"ID":"974dec77-4151-4c86-955b-61d7ca53da5d","Type":"ContainerStarted","Data":"58a7835e1309d5c6009b0f0f79f5fea1090741bc9142b1d442389ab8f765f009"} Jul 14 19:47:52.687862 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.687831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cpz46" event={"ID":"a1881a13-f947-4dbd-b8cf-f89d4a860b61","Type":"ContainerStarted","Data":"52a7c01a7232f27cc88687a09f7c0e6dc029ceb96813e9354e460e421aa7dfab"} Jul 14 19:47:52.689372 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.689351 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="d5660d22546d496cc857781c7b4baee2a4e3d527bdf339d6c7514871e0977b9c" exitCode=0 Jul 14 19:47:52.689469 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.689413 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"d5660d22546d496cc857781c7b4baee2a4e3d527bdf339d6c7514871e0977b9c"} Jul 14 19:47:52.691441 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.691417 2574 generic.go:356] "Generic (PLEG): container finished" podID="b31dba48238fdc286bfab6bd4db2f24f" containerID="b3054b47fc8d0f0bd8abecb32758a5bdc80ca728f1c980ef83f5f1495ca29e27" exitCode=0 Jul 14 19:47:52.691553 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.691520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" event={"ID":"b31dba48238fdc286bfab6bd4db2f24f","Type":"ContainerDied","Data":"b3054b47fc8d0f0bd8abecb32758a5bdc80ca728f1c980ef83f5f1495ca29e27"} Jul 14 19:47:52.693077 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.693053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" event={"ID":"43ba1ebae5d7ee21a4751f58c11fde6b","Type":"ContainerStarted","Data":"1282daebb596b722f9b5fb1c5a395979d4786b135f6b47c6bbfc5fa9ffd9dc1b"} Jul 14 19:47:52.702223 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.702180 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6ttml" podStartSLOduration=2.710378262 podStartE2EDuration="22.702168857s" totalImagesPullingTime="19.991790595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.701858309 +0000 UTC m=+22.546195208" watchObservedRunningTime="2026-07-14 19:47:52.702168857 +0000 UTC m=+22.546505730" Jul 14 19:47:52.743450 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.743271 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dzpk9" podStartSLOduration=2.985056925 podStartE2EDuration="22.743254173s" totalImagesPullingTime="19.758197248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.742672168 +0000 UTC m=+22.587009042" watchObservedRunningTime="2026-07-14 19:47:52.743254173 +0000 UTC m=+22.587591048" Jul 14 19:47:52.761663 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.761619 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dltdc" podStartSLOduration=3.736256396 podStartE2EDuration="22.761602575s" totalImagesPullingTime="19.025346179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.761401948 +0000 UTC m=+22.605738825" watchObservedRunningTime="2026-07-14 19:47:52.761602575 +0000 UTC m=+22.605939449" Jul 14 19:47:52.776220 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.776186 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vw69b" podStartSLOduration=7.238714988 podStartE2EDuration="22.776173245s" totalImagesPullingTime="15.537458257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.77550185 +0000 UTC m=+22.619838718" watchObservedRunningTime="2026-07-14 19:47:52.776173245 +0000 UTC m=+22.620510170" Jul 14 19:47:52.810813 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.810779 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-28.ec2.internal" podStartSLOduration=21.810767166 podStartE2EDuration="21.810767166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.810630993 +0000 UTC m=+22.654967871" watchObservedRunningTime="2026-07-14 19:47:52.810767166 +0000 UTC m=+22.655104040" Jul 14 19:47:52.830535 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:52.830475 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cpz46" podStartSLOduration=11.027452752 podStartE2EDuration="22.830461358s" totalImagesPullingTime="11.803008606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:52.830408162 +0000 UTC m=+22.674745036" watchObservedRunningTime="2026-07-14 19:47:52.830461358 +0000 UTC m=+22.674798231" Jul 14 19:47:53.311628 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.311598 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:53.312216 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.312197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:53.594390 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.594360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:53.594714 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.594396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:53.594714 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:53.594471 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:53.594714 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.594360 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:53.594714 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:53.594590 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:53.594714 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:53.594667 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:53.699435 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.699397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"8d8bc45468ab9bb8820765019a29f534d62069ae9462d9dec34bf658907b50c3"} Jul 14 19:47:53.699874 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.699506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"362d312c15df137e971686d2960fed05cc8f13ca9f2d3e774589c988f3a57008"} Jul 14 19:47:53.699874 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.699559 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"367897c77c27f471ed00d25f3a6b7aa7eef2046f85281721ec8c372c4e66dc99"} Jul 14 19:47:53.701047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.701020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-87hfz" event={"ID":"09afce7d-82ab-41b5-8c8d-8dbddc9a619e","Type":"ContainerStarted","Data":"2169a056824793e1c20640b9ea882828ad3dc40084c61bf1a2f19f62fbaeb78b"} Jul 14 19:47:53.703194 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.703164 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" event={"ID":"b31dba48238fdc286bfab6bd4db2f24f","Type":"ContainerStarted","Data":"b2b3f9014917ea241c1c9e3689b6bef44dc49821348695a37d35d0af19bdb2d0"} Jul 14 19:47:53.704190 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.704164 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:53.704717 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.704695 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cpz46" Jul 14 19:47:53.715663 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.715605 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-87hfz" podStartSLOduration=4.158262864 podStartE2EDuration="23.715591367s" totalImagesPullingTime="19.557328503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.715384392 +0000 UTC m=+23.559721267" watchObservedRunningTime="2026-07-14 19:47:53.715591367 +0000 UTC m=+23.559928275" Jul 14 19:47:53.750082 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:53.747727 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-28.ec2.internal" podStartSLOduration=22.747708861 podStartE2EDuration="22.747708861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:53.746756869 +0000 UTC m=+23.591093743" watchObservedRunningTime="2026-07-14 19:47:53.747708861 +0000 UTC m=+23.592045736" Jul 14 19:47:54.386239 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:54.386217 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:47:54.551463 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:54.551295 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:47:54.386233632Z","UUID":"320f72dc-bffe-42e8-8dd4-d221b6d99709","Handler":null,"Name":"","Endpoint":""} Jul 14 19:47:54.553243 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:54.553219 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:47:54.553422 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:54.553263 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:47:54.708907 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:54.708869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" event={"ID":"9a78e5d3-fc7d-4702-bb80-9a78631e5689","Type":"ContainerStarted","Data":"b6b831fdad50bce826ec6d4e4c50543559d024b4c30f74694c172ef650c7a4ce"} Jul 14 19:47:55.593968 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:55.593934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:55.594182 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:55.594045 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:55.594182 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:55.593934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:55.594182 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:55.593946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:55.594326 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:55.594175 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:55.594326 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:55.594245 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:55.715294 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:55.715247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"d46ae4180a913f6c9150710700fb3e46e62a612df8fe26729818063b4c763f9e"} Jul 14 19:47:56.720771 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:56.720727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" event={"ID":"9a78e5d3-fc7d-4702-bb80-9a78631e5689","Type":"ContainerStarted","Data":"f5c8412376a323c50739425fde70c470a7ec422089d036b75a34ba2409b05532"} Jul 14 19:47:56.739392 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:56.739342 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-567nc" podStartSLOduration=4.460958406 podStartE2EDuration="26.739324156s" totalImagesPullingTime="22.27836575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:56.73818093 +0000 UTC m=+26.582517808" watchObservedRunningTime="2026-07-14 19:47:56.739324156 +0000 UTC m=+26.583661028" Jul 14 19:47:57.593858 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:57.593821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:57.594043 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:57.593873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:57.594043 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:57.593821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:57.594043 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:57.593938 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:57.594043 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:57.594002 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:57.594212 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:57.594079 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:58.726762 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.726567 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="bbec4a3a8f97e6eb6839afd77552997b8358d480d3ae72eb4ed2e5d90464b460" exitCode=0 Jul 14 19:47:58.727423 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.726650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"bbec4a3a8f97e6eb6839afd77552997b8358d480d3ae72eb4ed2e5d90464b460"} Jul 14 19:47:58.730951 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.730926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" event={"ID":"3c642359-2a0f-438f-84eb-e1a59e6d2fb5","Type":"ContainerStarted","Data":"4e8dc4816254b2a1d14b6cca257c7ab83cf602365bf82564b729f5cdf802dead"} Jul 14 19:47:58.731247 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.731224 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:58.731359 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.731295 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:58.746575 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.746556 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:58.746705 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.746693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:47:58.776631 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:58.776577 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" podStartSLOduration=9.28812939 podStartE2EDuration="28.776562485s" totalImagesPullingTime="19.488433095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:58.776418108 +0000 UTC m=+28.620754981" watchObservedRunningTime="2026-07-14 19:47:58.776562485 +0000 UTC m=+28.620899358" Jul 14 19:47:59.594484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:59.594453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:47:59.594768 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:59.594580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:47:59.594768 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:59.594588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:47:59.594768 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:59.594606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:47:59.594768 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:59.594692 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:47:59.595020 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:47:59.594766 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:47:59.735931 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:47:59.734259 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 19:48:00.710705 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:00.710672 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:48:00.738257 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:00.738224 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="1dad2711e30108d70478b183c09779c36eb99f3e18159a0c80b8242328992255" exitCode=0 Jul 14 19:48:00.738634 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:00.738312 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"1dad2711e30108d70478b183c09779c36eb99f3e18159a0c80b8242328992255"} Jul 14 19:48:01.593507 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:01.593461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:01.593672 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:01.593468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:01.593672 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:01.593577 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:01.593800 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:01.593683 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:01.593800 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:01.593472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:01.593800 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:01.593786 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:01.742679 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:01.742468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerStarted","Data":"deb220244e5dfe0aa7fa9e48cc6768d6e0236afeacf8920c19308cfcd6503d43"} Jul 14 19:48:02.747291 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:02.747252 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="deb220244e5dfe0aa7fa9e48cc6768d6e0236afeacf8920c19308cfcd6503d43" exitCode=0 Jul 14 19:48:02.747682 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:02.747314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"deb220244e5dfe0aa7fa9e48cc6768d6e0236afeacf8920c19308cfcd6503d43"} Jul 14 19:48:03.321055 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.321021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:03.321244 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.321096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:03.321244 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.321180 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:48:03.321244 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.321180 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:48:03.321244 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.321239 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret podName:daa50ea0-e19e-4c48-9afc-adc079abf2d5 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:35.321223204 +0000 UTC m=+65.165560062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret") pod "global-pull-secret-syncer-8rvfr" (UID: "daa50ea0-e19e-4c48-9afc-adc079abf2d5") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:48:03.321440 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.321257 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs podName:99b20e35-bc5e-44f8-94de-2828be472ec0 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:35.321249438 +0000 UTC m=+65.165586296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs") pod "network-metrics-daemon-s5c8w" (UID: "99b20e35-bc5e-44f8-94de-2828be472ec0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:48:03.422418 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.422381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:03.422596 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.422540 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:48:03.422596 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.422568 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:48:03.422596 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.422580 2574 projected.go:196] Error preparing data for projected volume kube-api-access-tctnn for pod openshift-network-diagnostics/network-check-target-zvd8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:48:03.422757 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.422646 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn podName:d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f nodeName:}" failed. No retries permitted until 2026-07-14 19:48:35.422627285 +0000 UTC m=+65.266964151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tctnn" (UniqueName: "kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn") pod "network-check-target-zvd8p" (UID: "d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:48:03.593850 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.593768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:03.593850 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.593811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:03.594064 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:03.593871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:03.594064 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.593882 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:03.594064 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.593943 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:03.594064 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:03.594043 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:05.594537 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:05.594483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:05.594537 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:05.594536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:05.595135 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:05.594625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:05.595135 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:05.594633 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:05.595135 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:05.594771 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:05.595135 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:05.594853 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:07.594290 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:07.594260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:07.594709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:07.594264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:07.594709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:07.594428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:07.594709 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:07.594386 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:07.594709 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:07.594536 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:07.594709 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:07.594628 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:07.762241 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:07.762210 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="c4cd176872dfa055de512496636cb8aad8ae157b6fc8fe9cb1005d396a472192" exitCode=0 Jul 14 19:48:07.762395 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:07.762265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"c4cd176872dfa055de512496636cb8aad8ae157b6fc8fe9cb1005d396a472192"} Jul 14 19:48:08.266253 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.266218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rvfr"] Jul 14 19:48:08.266406 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.266327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:08.266473 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:08.266434 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:08.269155 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.269130 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvd8p"] Jul 14 19:48:08.269303 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.269211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:08.273155 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:08.269699 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:08.273155 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.271408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5c8w"] Jul 14 19:48:08.273155 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.271615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:08.273155 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:08.271889 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:08.767966 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.767933 2574 generic.go:356] "Generic (PLEG): container finished" podID="80dfc55c-0e33-4820-b465-87fa724e7757" containerID="e430bae73255d33338a938df6c562c5ee88490ab4dbeb64c974c693b67ee78d5" exitCode=0 Jul 14 19:48:08.768382 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:08.767997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerDied","Data":"e430bae73255d33338a938df6c562c5ee88490ab4dbeb64c974c693b67ee78d5"} Jul 14 19:48:09.594347 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:09.594163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:09.594581 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:09.594171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:09.594581 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:09.594437 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:09.594581 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:09.594195 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:09.594581 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:09.594538 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:09.594581 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:09.594577 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:09.773561 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:09.773528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tpn29" event={"ID":"80dfc55c-0e33-4820-b465-87fa724e7757","Type":"ContainerStarted","Data":"9c1ab31fddf78e7fcfe35f0fa01ca1665c530c0f11277d9fdf49bd71a912b06d"} Jul 14 19:48:09.795986 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:09.795945 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tpn29" podStartSLOduration=16.768896208 podStartE2EDuration="39.795930445s" totalImagesPullingTime="23.027034237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:09.795350268 +0000 UTC m=+39.639687142" watchObservedRunningTime="2026-07-14 19:48:09.795930445 +0000 UTC m=+39.640267319" Jul 14 19:48:11.593821 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:11.593783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:11.593821 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:11.593809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:11.594280 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:11.593830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:11.594280 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:11.593923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s5c8w" podUID="99b20e35-bc5e-44f8-94de-2828be472ec0" Jul 14 19:48:11.594280 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:11.593971 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8rvfr" podUID="daa50ea0-e19e-4c48-9afc-adc079abf2d5" Jul 14 19:48:11.594280 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:11.594045 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zvd8p" podUID="d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f" Jul 14 19:48:12.043268 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.043238 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-28.ec2.internal" event="NodeReady" Jul 14 19:48:12.043449 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.043353 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:48:12.077247 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.077219 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-849757d945-bdmfj"] Jul 14 19:48:12.096676 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.096643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.098532 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.098484 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bczlc"] Jul 14 19:48:12.099747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.099506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 19:48:12.099747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.099536 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 19:48:12.099997 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.099981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-65424\"" Jul 14 19:48:12.100633 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.100615 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 19:48:12.105624 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.105609 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 19:48:12.114650 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.114633 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849757d945-bdmfj"] Jul 14 19:48:12.114743 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.114659 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jdlcb"] Jul 14 19:48:12.115231 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.114833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.118179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.118058 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-949mk\"" Jul 14 19:48:12.118179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.118075 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:48:12.118179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.118098 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:48:12.138638 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.138616 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdlcb"] Jul 14 19:48:12.138761 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.138660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bczlc"] Jul 14 19:48:12.138824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.138768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.141320 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.141283 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:48:12.141320 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.141301 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:48:12.141463 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.141438 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-stdzp\"" Jul 14 19:48:12.141463 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.141454 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:48:12.141463 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.141458 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:48:12.192112 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-tls\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192219 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-trusted-ca\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192219 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a352d-44e2-423c-b0af-758ed4742235-metrics-tls\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.192219 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-installation-pull-secrets\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192219 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-image-registry-private-configuration\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192219 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-kube-api-access-c7kzm\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0a352d-44e2-423c-b0af-758ed4742235-tmp-dir\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5155ceaf-504a-4a44-9fda-ab319c15539a-ca-trust-extracted\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-bound-sa-token\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrmv\" (UniqueName: \"kubernetes.io/projected/7c0a352d-44e2-423c-b0af-758ed4742235-kube-api-access-dfrmv\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0a352d-44e2-423c-b0af-758ed4742235-config-volume\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.192461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.192457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-certificates\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.194890 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.194868 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8vpfn"] Jul 14 19:48:12.218256 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.218223 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vpfn"] Jul 14 19:48:12.218334 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.218311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.220875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.220858 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:48:12.220875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.220869 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:48:12.221064 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.220919 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nfcd5\"" Jul 14 19:48:12.221167 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.221151 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:48:12.292753 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ef62b39-b6dc-4165-80ef-e7aa6163f239-data-volume\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-installation-pull-secrets\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-image-registry-private-configuration\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-kube-api-access-c7kzm\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0a352d-44e2-423c-b0af-758ed4742235-tmp-dir\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5155ceaf-504a-4a44-9fda-ab319c15539a-ca-trust-extracted\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-bound-sa-token\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.292883 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrmv\" (UniqueName: \"kubernetes.io/projected/7c0a352d-44e2-423c-b0af-758ed4742235-kube-api-access-dfrmv\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0a352d-44e2-423c-b0af-758ed4742235-config-volume\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-certificates\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-tls\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.292996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9l4\" (UniqueName: \"kubernetes.io/projected/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-api-access-mp9l4\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-trusted-ca\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a352d-44e2-423c-b0af-758ed4742235-metrics-tls\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.293215 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ef62b39-b6dc-4165-80ef-e7aa6163f239-crio-socket\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.293618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293369 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0a352d-44e2-423c-b0af-758ed4742235-tmp-dir\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.293618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ef62b39-b6dc-4165-80ef-e7aa6163f239-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.293618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5155ceaf-504a-4a44-9fda-ab319c15539a-ca-trust-extracted\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.293755 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.293628 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c0a352d-44e2-423c-b0af-758ed4742235-config-volume\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.294283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.294259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-certificates\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.294469 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.294449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5155ceaf-504a-4a44-9fda-ab319c15539a-trusted-ca\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.297190 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.297167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c0a352d-44e2-423c-b0af-758ed4742235-metrics-tls\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.297267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.297197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-image-registry-private-configuration\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.297267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.297249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5155ceaf-504a-4a44-9fda-ab319c15539a-installation-pull-secrets\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.297332 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.297277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-registry-tls\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.303712 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.303691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-bound-sa-token\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.303983 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.303956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrmv\" (UniqueName: \"kubernetes.io/projected/7c0a352d-44e2-423c-b0af-758ed4742235-kube-api-access-dfrmv\") pod \"dns-default-bczlc\" (UID: \"7c0a352d-44e2-423c-b0af-758ed4742235\") " pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.304115 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.304093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/5155ceaf-504a-4a44-9fda-ab319c15539a-kube-api-access-c7kzm\") pod \"image-registry-849757d945-bdmfj\" (UID: \"5155ceaf-504a-4a44-9fda-ab319c15539a\") " pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.394212 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zgs\" (UniqueName: \"kubernetes.io/projected/edcb30b6-4adb-4e40-84e2-1ac353237d17-kube-api-access-v9zgs\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9l4\" (UniqueName: \"kubernetes.io/projected/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-api-access-mp9l4\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ef62b39-b6dc-4165-80ef-e7aa6163f239-crio-socket\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ef62b39-b6dc-4165-80ef-e7aa6163f239-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ef62b39-b6dc-4165-80ef-e7aa6163f239-data-volume\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394329 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb30b6-4adb-4e40-84e2-1ac353237d17-cert\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.394626 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ef62b39-b6dc-4165-80ef-e7aa6163f239-crio-socket\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394677 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ef62b39-b6dc-4165-80ef-e7aa6163f239-data-volume\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.394865 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.394836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.396611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.396593 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ef62b39-b6dc-4165-80ef-e7aa6163f239-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.402876 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.402855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9l4\" (UniqueName: \"kubernetes.io/projected/1ef62b39-b6dc-4165-80ef-e7aa6163f239-kube-api-access-mp9l4\") pod \"insights-runtime-extractor-jdlcb\" (UID: \"1ef62b39-b6dc-4165-80ef-e7aa6163f239\") " pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.407828 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.407810 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.425200 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.425180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:12.446451 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.446434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdlcb" Jul 14 19:48:12.495305 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.495277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zgs\" (UniqueName: \"kubernetes.io/projected/edcb30b6-4adb-4e40-84e2-1ac353237d17-kube-api-access-v9zgs\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.495416 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.495347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb30b6-4adb-4e40-84e2-1ac353237d17-cert\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.499148 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.499122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edcb30b6-4adb-4e40-84e2-1ac353237d17-cert\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.504988 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.504957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zgs\" (UniqueName: \"kubernetes.io/projected/edcb30b6-4adb-4e40-84e2-1ac353237d17-kube-api-access-v9zgs\") pod \"ingress-canary-8vpfn\" (UID: \"edcb30b6-4adb-4e40-84e2-1ac353237d17\") " pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.527224 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.527203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vpfn" Jul 14 19:48:12.632935 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.632769 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bczlc"] Jul 14 19:48:12.636322 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:12.636295 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0a352d_44e2_423c_b0af_758ed4742235.slice/crio-d5adaffeb7e17d0fdfc4adafa96c77f7801e38f4d73198ae54c30e02d2c052ac WatchSource:0}: Error finding container d5adaffeb7e17d0fdfc4adafa96c77f7801e38f4d73198ae54c30e02d2c052ac: Status 404 returned error can't find the container with id d5adaffeb7e17d0fdfc4adafa96c77f7801e38f4d73198ae54c30e02d2c052ac Jul 14 19:48:12.648396 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.647612 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdlcb"] Jul 14 19:48:12.649760 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.649080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849757d945-bdmfj"] Jul 14 19:48:12.650573 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:12.650550 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5155ceaf_504a_4a44_9fda_ab319c15539a.slice/crio-9d2db001a81b1805561e2e2764fe830e4f729af5e2a336098574f546d25696a0 WatchSource:0}: Error finding container 9d2db001a81b1805561e2e2764fe830e4f729af5e2a336098574f546d25696a0: Status 404 returned error can't find the container with id 9d2db001a81b1805561e2e2764fe830e4f729af5e2a336098574f546d25696a0 Jul 14 19:48:12.651076 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:12.651049 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef62b39_b6dc_4165_80ef_e7aa6163f239.slice/crio-42e06acbb76819ce411b55da80ba831e8017ece3b0b07426516db1cc2c1e7d60 WatchSource:0}: Error finding container 42e06acbb76819ce411b55da80ba831e8017ece3b0b07426516db1cc2c1e7d60: Status 404 returned error can't find the container with id 42e06acbb76819ce411b55da80ba831e8017ece3b0b07426516db1cc2c1e7d60 Jul 14 19:48:12.695035 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.695006 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vpfn"] Jul 14 19:48:12.699000 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:12.698976 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcb30b6_4adb_4e40_84e2_1ac353237d17.slice/crio-43eb785db2879578e000050f16088d654c39959a93db02f3189ba4e6f4570d07 WatchSource:0}: Error finding container 43eb785db2879578e000050f16088d654c39959a93db02f3189ba4e6f4570d07: Status 404 returned error can't find the container with id 43eb785db2879578e000050f16088d654c39959a93db02f3189ba4e6f4570d07 Jul 14 19:48:12.781151 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.781118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vpfn" event={"ID":"edcb30b6-4adb-4e40-84e2-1ac353237d17","Type":"ContainerStarted","Data":"43eb785db2879578e000050f16088d654c39959a93db02f3189ba4e6f4570d07"} Jul 14 19:48:12.782539 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.782513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849757d945-bdmfj" event={"ID":"5155ceaf-504a-4a44-9fda-ab319c15539a","Type":"ContainerStarted","Data":"ea14e4e0c5d9cf4f69bc4daea791f123353fc0410455a7ef8c6eb75e4ca9fd08"} Jul 14 19:48:12.782650 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.782556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849757d945-bdmfj" event={"ID":"5155ceaf-504a-4a44-9fda-ab319c15539a","Type":"ContainerStarted","Data":"9d2db001a81b1805561e2e2764fe830e4f729af5e2a336098574f546d25696a0"} Jul 14 19:48:12.782650 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.782598 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:12.783627 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.783604 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdlcb" event={"ID":"1ef62b39-b6dc-4165-80ef-e7aa6163f239","Type":"ContainerStarted","Data":"42e06acbb76819ce411b55da80ba831e8017ece3b0b07426516db1cc2c1e7d60"} Jul 14 19:48:12.784528 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.784508 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bczlc" event={"ID":"7c0a352d-44e2-423c-b0af-758ed4742235","Type":"ContainerStarted","Data":"d5adaffeb7e17d0fdfc4adafa96c77f7801e38f4d73198ae54c30e02d2c052ac"} Jul 14 19:48:12.803561 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:12.803526 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-849757d945-bdmfj" podStartSLOduration=6.80348416 podStartE2EDuration="6.80348416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:12.802411248 +0000 UTC m=+42.646748114" watchObservedRunningTime="2026-07-14 19:48:12.80348416 +0000 UTC m=+42.647821033" Jul 14 19:48:13.594438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.594401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:13.594888 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.594837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:13.595346 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.595331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:13.597379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.597353 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:48:13.598609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.598586 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:48:13.598840 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.598818 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:48:13.599006 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.598991 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45mdm\"" Jul 14 19:48:13.599223 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.599208 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:48:13.599436 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:13.599421 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6fln\"" Jul 14 19:48:19.808811 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.808606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdlcb" event={"ID":"1ef62b39-b6dc-4165-80ef-e7aa6163f239","Type":"ContainerStarted","Data":"5ad4cd20fd4460ea8abbdb5abff74a3ca2fd00a29060a656380b234d28c0532a"} Jul 14 19:48:19.810449 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.810422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bczlc" event={"ID":"7c0a352d-44e2-423c-b0af-758ed4742235","Type":"ContainerStarted","Data":"f7e457827a783df4754e4409b82bd68ba1525126ed4a8bb1f5bfb5f2c0df0731"} Jul 14 19:48:19.810587 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.810482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bczlc" event={"ID":"7c0a352d-44e2-423c-b0af-758ed4742235","Type":"ContainerStarted","Data":"40e91ff7c3ca9b3ba1e823a4b88d1ee98a8cc2017cb3024a3e3ca75f92762c56"} Jul 14 19:48:19.810684 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.810660 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:19.811923 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.811890 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vpfn" event={"ID":"edcb30b6-4adb-4e40-84e2-1ac353237d17","Type":"ContainerStarted","Data":"0f608bc947d4ebfbf7e2ab2adfc3745c7a7e4e52d813898069181db1e0391308"} Jul 14 19:48:19.830184 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.830146 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bczlc" podStartSLOduration=1.53990734 podStartE2EDuration="7.830133807s" totalImagesPullingTime="6.290226467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:19.828265046 +0000 UTC m=+49.672601922" watchObservedRunningTime="2026-07-14 19:48:19.830133807 +0000 UTC m=+49.674470682" Jul 14 19:48:19.843797 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:19.843748 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8vpfn" podStartSLOduration=1.608754654 podStartE2EDuration="7.843736156s" totalImagesPullingTime="6.234981502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:19.842972796 +0000 UTC m=+49.687309671" watchObservedRunningTime="2026-07-14 19:48:19.843736156 +0000 UTC m=+49.688073033" Jul 14 19:48:20.816556 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:20.816517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdlcb" event={"ID":"1ef62b39-b6dc-4165-80ef-e7aa6163f239","Type":"ContainerStarted","Data":"716f8c0e2a85c737986e9d8aff36c8e10e69347d8389d78ed0c8287a12740d21"} Jul 14 19:48:21.396985 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.396947 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xsss5"] Jul 14 19:48:21.412407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.412382 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xsss5"] Jul 14 19:48:21.412592 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.412480 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.416321 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416298 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:48:21.416451 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416372 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:48:21.416559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416538 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 19:48:21.416559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416298 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 19:48:21.416924 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416783 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:48:21.416924 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.416803 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vc5qr\"" Jul 14 19:48:21.570852 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.570817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2eef7cb-9465-4f2e-b48e-422f568529bf-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.571040 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.570870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.571040 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.570934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fgc\" (UniqueName: \"kubernetes.io/projected/f2eef7cb-9465-4f2e-b48e-422f568529bf-kube-api-access-b4fgc\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.571040 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.571030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.671473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.671384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.671473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.671445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2eef7cb-9465-4f2e-b48e-422f568529bf-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.671726 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.671502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.671726 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.671636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fgc\" (UniqueName: \"kubernetes.io/projected/f2eef7cb-9465-4f2e-b48e-422f568529bf-kube-api-access-b4fgc\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.672228 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.672209 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2eef7cb-9465-4f2e-b48e-422f568529bf-metrics-client-ca\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.675899 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.675872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.676001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.675921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2eef7cb-9465-4f2e-b48e-422f568529bf-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.680369 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.680349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fgc\" (UniqueName: \"kubernetes.io/projected/f2eef7cb-9465-4f2e-b48e-422f568529bf-kube-api-access-b4fgc\") pod \"prometheus-operator-64785b6675-xsss5\" (UID: \"f2eef7cb-9465-4f2e-b48e-422f568529bf\") " pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.723255 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.723229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" Jul 14 19:48:21.873048 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:21.872833 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-xsss5"] Jul 14 19:48:23.032933 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:23.032896 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2eef7cb_9465_4f2e_b48e_422f568529bf.slice/crio-5a6ed2ddd8feb26ad18a51c768063c2149f221e888c81785b6fcbc62a7eebc92 WatchSource:0}: Error finding container 5a6ed2ddd8feb26ad18a51c768063c2149f221e888c81785b6fcbc62a7eebc92: Status 404 returned error can't find the container with id 5a6ed2ddd8feb26ad18a51c768063c2149f221e888c81785b6fcbc62a7eebc92 Jul 14 19:48:23.831418 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:23.831383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdlcb" event={"ID":"1ef62b39-b6dc-4165-80ef-e7aa6163f239","Type":"ContainerStarted","Data":"1b49d65b799cf3e77548b5df978c326a5da4ba67062eb1770144fe4efbcfc6e6"} Jul 14 19:48:23.832467 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:23.832445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" event={"ID":"f2eef7cb-9465-4f2e-b48e-422f568529bf","Type":"ContainerStarted","Data":"5a6ed2ddd8feb26ad18a51c768063c2149f221e888c81785b6fcbc62a7eebc92"} Jul 14 19:48:23.849163 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:23.849117 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jdlcb" podStartSLOduration=1.364390493 podStartE2EDuration="11.849100055s" totalImagesPullingTime="10.484709562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:48:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:23.847827323 +0000 UTC m=+53.692164191" watchObservedRunningTime="2026-07-14 19:48:23.849100055 +0000 UTC m=+53.693436931" Jul 14 19:48:24.840579 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:24.840365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" event={"ID":"f2eef7cb-9465-4f2e-b48e-422f568529bf","Type":"ContainerStarted","Data":"1dfa882ba1dcbeac8e75d2bf6b75000ee5067943c2a3bfb37e815e4d0e12e42d"} Jul 14 19:48:25.844753 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:25.844709 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" event={"ID":"f2eef7cb-9465-4f2e-b48e-422f568529bf","Type":"ContainerStarted","Data":"05c5e78531642ff49c9763d7f7aaad8b420754df726baa3a22f6dfda47271f14"} Jul 14 19:48:25.862292 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:25.862249 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-xsss5" podStartSLOduration=3.130766678 podStartE2EDuration="4.862231465s" totalImagesPullingTime="1.731464787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:25.86093209 +0000 UTC m=+55.705268964" watchObservedRunningTime="2026-07-14 19:48:25.862231465 +0000 UTC m=+55.706568338" Jul 14 19:48:27.768598 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.768563 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6wgd4"] Jul 14 19:48:27.771782 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.771762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.774159 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.774126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:48:27.774270 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.774189 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4l8n4\"" Jul 14 19:48:27.774270 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.774201 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:48:27.774270 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.774215 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:48:27.781163 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.781144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jhq4l"] Jul 14 19:48:27.784235 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.784216 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.786417 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.786397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kl9s8\"" Jul 14 19:48:27.786788 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.786768 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 19:48:27.786891 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.786842 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 19:48:27.786891 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.786834 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 19:48:27.797303 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.797279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jhq4l"] Jul 14 19:48:27.814331 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-sys\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-metrics-client-ca\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-tls\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kq7\" (UniqueName: \"kubernetes.io/projected/20e53405-752c-4424-8bd6-510cb766d7f9-kube-api-access-c6kq7\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.814530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814595 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-textfile\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-root\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ck4c\" (UniqueName: \"kubernetes.io/projected/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-api-access-5ck4c\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-wtmp\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.814733 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.814714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/838f0de2-ba0f-4963-af68-e57a89cd2066-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915263 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915263 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ck4c\" (UniqueName: \"kubernetes.io/projected/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-api-access-5ck4c\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-wtmp\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/838f0de2-ba0f-4963-af68-e57a89cd2066-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-sys\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:27.915370 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-metrics-client-ca\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915433 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:27.915428 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls podName:838f0de2-ba0f-4963-af68-e57a89cd2066 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:28.415410411 +0000 UTC m=+58.259747263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-jhq4l" (UID: "838f0de2-ba0f-4963-af68-e57a89cd2066") : secret "kube-state-metrics-tls" not found Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-wtmp\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915476 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-sys\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-tls\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kq7\" (UniqueName: \"kubernetes.io/projected/20e53405-752c-4424-8bd6-510cb766d7f9-kube-api-access-c6kq7\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-textfile\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-root\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.915827 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.915794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/838f0de2-ba0f-4963-af68-e57a89cd2066-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.916296 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-metrics-client-ca\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.916296 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.916296 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916083 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-textfile\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.916296 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20e53405-752c-4424-8bd6-510cb766d7f9-root\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.916509 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.916571 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.916554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f0de2-ba0f-4963-af68-e57a89cd2066-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.919697 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.919676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-tls\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.919697 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.919682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20e53405-752c-4424-8bd6-510cb766d7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.919819 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.919801 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:27.923987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.923970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kq7\" (UniqueName: \"kubernetes.io/projected/20e53405-752c-4424-8bd6-510cb766d7f9-kube-api-access-c6kq7\") pod \"node-exporter-6wgd4\" (UID: \"20e53405-752c-4424-8bd6-510cb766d7f9\") " pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:27.924313 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:27.924291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ck4c\" (UniqueName: \"kubernetes.io/projected/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-api-access-5ck4c\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:28.082195 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.082124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6wgd4" Jul 14 19:48:28.093990 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:28.093950 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e53405_752c_4424_8bd6_510cb766d7f9.slice/crio-3ea2efa7b7c514ca83d4cbc9f4a6bea4e77a53caf5c21d384137799f2149e04a WatchSource:0}: Error finding container 3ea2efa7b7c514ca83d4cbc9f4a6bea4e77a53caf5c21d384137799f2149e04a: Status 404 returned error can't find the container with id 3ea2efa7b7c514ca83d4cbc9f4a6bea4e77a53caf5c21d384137799f2149e04a Jul 14 19:48:28.419912 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.419829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:28.422327 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.422304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/838f0de2-ba0f-4963-af68-e57a89cd2066-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-jhq4l\" (UID: \"838f0de2-ba0f-4963-af68-e57a89cd2066\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:28.692148 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.692124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" Jul 14 19:48:28.836007 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.835867 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-jhq4l"] Jul 14 19:48:28.838524 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:28.838468 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f0de2_ba0f_4963_af68_e57a89cd2066.slice/crio-90f1bdc9d7127b8876eb8a2fd7dac244545187a0dfd69e3c22d5e9e1aebd6631 WatchSource:0}: Error finding container 90f1bdc9d7127b8876eb8a2fd7dac244545187a0dfd69e3c22d5e9e1aebd6631: Status 404 returned error can't find the container with id 90f1bdc9d7127b8876eb8a2fd7dac244545187a0dfd69e3c22d5e9e1aebd6631 Jul 14 19:48:28.855453 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.855418 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6wgd4" event={"ID":"20e53405-752c-4424-8bd6-510cb766d7f9","Type":"ContainerStarted","Data":"3ea2efa7b7c514ca83d4cbc9f4a6bea4e77a53caf5c21d384137799f2149e04a"} Jul 14 19:48:28.856557 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:28.856536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" event={"ID":"838f0de2-ba0f-4963-af68-e57a89cd2066","Type":"ContainerStarted","Data":"90f1bdc9d7127b8876eb8a2fd7dac244545187a0dfd69e3c22d5e9e1aebd6631"} Jul 14 19:48:29.819510 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:29.819465 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bczlc" Jul 14 19:48:31.754021 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.753993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-68t8w" Jul 14 19:48:31.868296 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.868258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" event={"ID":"838f0de2-ba0f-4963-af68-e57a89cd2066","Type":"ContainerStarted","Data":"b7e51e9beb70a9277b2cc0e5d9005a100c66f5633ddfd6ce06920a49b35d11f8"} Jul 14 19:48:31.868478 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.868313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" event={"ID":"838f0de2-ba0f-4963-af68-e57a89cd2066","Type":"ContainerStarted","Data":"f357d35b1696a92ee9748306778c5ab633da7b084f9fa7cd3f44a7903647b4b0"} Jul 14 19:48:31.868478 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.868334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" event={"ID":"838f0de2-ba0f-4963-af68-e57a89cd2066","Type":"ContainerStarted","Data":"daede3086e3c90e892d02665bf4532cf880924b19bd16f3ae0287bc4b2780078"} Jul 14 19:48:31.869747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.869719 2574 generic.go:356] "Generic (PLEG): container finished" podID="20e53405-752c-4424-8bd6-510cb766d7f9" containerID="2d297a4b433b2e09deae1f6a26c50a4715824c59cdf1353c7e55e0d42ca5bede" exitCode=0 Jul 14 19:48:31.869857 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.869750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6wgd4" event={"ID":"20e53405-752c-4424-8bd6-510cb766d7f9","Type":"ContainerDied","Data":"2d297a4b433b2e09deae1f6a26c50a4715824c59cdf1353c7e55e0d42ca5bede"} Jul 14 19:48:31.887256 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:31.887202 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-jhq4l" podStartSLOduration=2.838531231 podStartE2EDuration="4.887179417s" totalImagesPullingTime="2.048648186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:31.886077682 +0000 UTC m=+61.730414558" watchObservedRunningTime="2026-07-14 19:48:31.887179417 +0000 UTC m=+61.731516292" Jul 14 19:48:32.195501 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.195463 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b55f44bb6-fqpkh"] Jul 14 19:48:32.198503 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.198468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.201346 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201319 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 19:48:32.201437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 19:48:32.201437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 19:48:32.201601 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201319 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 19:48:32.201601 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f2c49h2nluc1q\"" Jul 14 19:48:32.201601 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.201367 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mrszq\"" Jul 14 19:48:32.208444 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.208415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b55f44bb6-fqpkh"] Jul 14 19:48:32.248744 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mmt\" (UniqueName: \"kubernetes.io/projected/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-kube-api-access-x9mmt\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248853 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-client-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248853 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-tls\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248853 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-metrics-server-audit-profiles\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248948 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-client-certs\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248948 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.248948 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.248938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-audit-log\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.349755 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.349725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.349868 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.349759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-audit-log\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.349908 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.349887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mmt\" (UniqueName: \"kubernetes.io/projected/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-kube-api-access-x9mmt\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.349972 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.349957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-client-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.350003 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.349992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-tls\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.350033 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.350019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-metrics-server-audit-profiles\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.350107 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.350086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-client-certs\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.350178 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.350160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-audit-log\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.350588 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.350560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.351199 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.351172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-metrics-server-audit-profiles\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.352705 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.352683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-client-ca-bundle\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.352804 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.352675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-client-certs\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.352804 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.352717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-secret-metrics-server-tls\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.358767 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.358749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mmt\" (UniqueName: \"kubernetes.io/projected/9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d-kube-api-access-x9mmt\") pod \"metrics-server-6b55f44bb6-fqpkh\" (UID: \"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d\") " pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.507969 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.507888 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:32.539916 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.539881 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb"] Jul 14 19:48:32.544578 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.544557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.547145 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.546918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 19:48:32.547145 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.546981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-47flk\"" Jul 14 19:48:32.553805 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.552118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.553805 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.552162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwnw\" (UniqueName: \"kubernetes.io/projected/09c295ef-6803-4875-927d-63c8e38a9a61-kube-api-access-tzwnw\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.554364 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.554346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb"] Jul 14 19:48:32.646637 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.646613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b55f44bb6-fqpkh"] Jul 14 19:48:32.648814 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:32.648794 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2eb0ac_8b8f_429e_a36b_2f0c572d1a1d.slice/crio-7e8c53e65eb8a66f3ec19981b1e3217957e3dcef701db1f56c10c3967ffa3510 WatchSource:0}: Error finding container 7e8c53e65eb8a66f3ec19981b1e3217957e3dcef701db1f56c10c3967ffa3510: Status 404 returned error can't find the container with id 7e8c53e65eb8a66f3ec19981b1e3217957e3dcef701db1f56c10c3967ffa3510 Jul 14 19:48:32.653443 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.653423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.653534 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.653453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwnw\" (UniqueName: \"kubernetes.io/projected/09c295ef-6803-4875-927d-63c8e38a9a61-kube-api-access-tzwnw\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.653605 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:32.653586 2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 14 19:48:32.653676 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:48:32.653665 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert podName:09c295ef-6803-4875-927d-63c8e38a9a61 nodeName:}" failed. No retries permitted until 2026-07-14 19:48:33.153642724 +0000 UTC m=+62.997979596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-7hdgb" (UID: "09c295ef-6803-4875-927d-63c8e38a9a61") : secret "monitoring-plugin-cert" not found Jul 14 19:48:32.661441 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.661419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwnw\" (UniqueName: \"kubernetes.io/projected/09c295ef-6803-4875-927d-63c8e38a9a61-kube-api-access-tzwnw\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:32.874609 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.874526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" event={"ID":"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d","Type":"ContainerStarted","Data":"7e8c53e65eb8a66f3ec19981b1e3217957e3dcef701db1f56c10c3967ffa3510"} Jul 14 19:48:32.880512 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.880460 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6wgd4" event={"ID":"20e53405-752c-4424-8bd6-510cb766d7f9","Type":"ContainerStarted","Data":"62ae6e05a10fb35238189b38a97de4a6f12e2ae69d9fb38244466dd48747d6c4"} Jul 14 19:48:32.880641 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.880569 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6wgd4" event={"ID":"20e53405-752c-4424-8bd6-510cb766d7f9","Type":"ContainerStarted","Data":"1376bbb4f181841d8fee830c1e314070a29da927390596c51ea10dbc5664004a"} Jul 14 19:48:32.900846 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:32.900805 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6wgd4" podStartSLOduration=3.019345099 podStartE2EDuration="5.900792116s" totalImagesPullingTime="2.795395009s" totalInitContainerRuntime="86.052008ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:32.899882343 +0000 UTC m=+62.744219219" watchObservedRunningTime="2026-07-14 19:48:32.900792116 +0000 UTC m=+62.745128989" Jul 14 19:48:33.158754 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.158666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:33.161593 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.161559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09c295ef-6803-4875-927d-63c8e38a9a61-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7hdgb\" (UID: \"09c295ef-6803-4875-927d-63c8e38a9a61\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:33.456363 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.456327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:33.585345 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.585321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb"] Jul 14 19:48:33.587518 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:33.587473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c295ef_6803_4875_927d_63c8e38a9a61.slice/crio-5594d94c0550c63f1c883cb6f84dba0af8cc91399a892289770be0c664f67ff1 WatchSource:0}: Error finding container 5594d94c0550c63f1c883cb6f84dba0af8cc91399a892289770be0c664f67ff1: Status 404 returned error can't find the container with id 5594d94c0550c63f1c883cb6f84dba0af8cc91399a892289770be0c664f67ff1 Jul 14 19:48:33.800293 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.800210 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-849757d945-bdmfj" Jul 14 19:48:33.888421 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:33.888347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" event={"ID":"09c295ef-6803-4875-927d-63c8e38a9a61","Type":"ContainerStarted","Data":"5594d94c0550c63f1c883cb6f84dba0af8cc91399a892289770be0c664f67ff1"} Jul 14 19:48:34.892897 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:34.892861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" event={"ID":"9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d","Type":"ContainerStarted","Data":"6c30989b3ac4a3196caf79a6c2f86cbc663e5224f0a844153d7edc4a4181714c"} Jul 14 19:48:34.914809 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:34.914758 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" podStartSLOduration=1.367924013 podStartE2EDuration="2.914744348s" totalImagesPullingTime="1.546820335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:34.913721593 +0000 UTC m=+64.758058470" watchObservedRunningTime="2026-07-14 19:48:34.914744348 +0000 UTC m=+64.759081222" Jul 14 19:48:35.379254 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.379210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:35.379445 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.379301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:35.382385 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.382358 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:48:35.383020 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.382994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:48:35.392591 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.392564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daa50ea0-e19e-4c48-9afc-adc079abf2d5-original-pull-secret\") pod \"global-pull-secret-syncer-8rvfr\" (UID: \"daa50ea0-e19e-4c48-9afc-adc079abf2d5\") " pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:35.392699 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.392634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99b20e35-bc5e-44f8-94de-2828be472ec0-metrics-certs\") pod \"network-metrics-daemon-s5c8w\" (UID: \"99b20e35-bc5e-44f8-94de-2828be472ec0\") " pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:35.480475 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.480438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:35.483014 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.482993 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:48:35.493022 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.493004 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:48:35.504763 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.504738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctnn\" (UniqueName: \"kubernetes.io/projected/d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f-kube-api-access-tctnn\") pod \"network-check-target-zvd8p\" (UID: \"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f\") " pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:35.516807 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.516783 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-45mdm\"" Jul 14 19:48:35.524931 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.524912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:35.527189 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.527166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8rvfr" Jul 14 19:48:35.536253 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.536237 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6fln\"" Jul 14 19:48:35.544639 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.544619 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s5c8w" Jul 14 19:48:35.881996 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.881967 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s5c8w"] Jul 14 19:48:35.885449 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:35.885408 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b20e35_bc5e_44f8_94de_2828be472ec0.slice/crio-ca4ef5acb33725f7e4bfbc47f426f7008e6266a4c29aa814b9403c551a579315 WatchSource:0}: Error finding container ca4ef5acb33725f7e4bfbc47f426f7008e6266a4c29aa814b9403c551a579315: Status 404 returned error can't find the container with id ca4ef5acb33725f7e4bfbc47f426f7008e6266a4c29aa814b9403c551a579315 Jul 14 19:48:35.897015 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.896987 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" event={"ID":"09c295ef-6803-4875-927d-63c8e38a9a61","Type":"ContainerStarted","Data":"9217f5370a9c9f0207fedf7fd69994b9ef4e6909849d5eb52ade726c74a50227"} Jul 14 19:48:35.897362 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.897184 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:35.898462 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.898436 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c8w" event={"ID":"99b20e35-bc5e-44f8-94de-2828be472ec0","Type":"ContainerStarted","Data":"ca4ef5acb33725f7e4bfbc47f426f7008e6266a4c29aa814b9403c551a579315"} Jul 14 19:48:35.902284 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.902263 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" Jul 14 19:48:35.914432 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:35.914396 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7hdgb" podStartSLOduration=1.72017061 podStartE2EDuration="3.91438156s" totalImagesPullingTime="2.19421095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:35.912998186 +0000 UTC m=+65.757335085" watchObservedRunningTime="2026-07-14 19:48:35.91438156 +0000 UTC m=+65.758718431" Jul 14 19:48:36.107844 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:36.107751 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8rvfr"] Jul 14 19:48:36.108738 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:36.108715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zvd8p"] Jul 14 19:48:36.110199 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:36.110170 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58b8886_eeeb_4add_bde3_c9b3a4bc2d7f.slice/crio-99aa4c5c5f0924e7e259480b25b380ea9083bbad98c957d4dcaf4ce1bfe0cda5 WatchSource:0}: Error finding container 99aa4c5c5f0924e7e259480b25b380ea9083bbad98c957d4dcaf4ce1bfe0cda5: Status 404 returned error can't find the container with id 99aa4c5c5f0924e7e259480b25b380ea9083bbad98c957d4dcaf4ce1bfe0cda5 Jul 14 19:48:36.110788 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:48:36.110771 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa50ea0_e19e_4c48_9afc_adc079abf2d5.slice/crio-e5e105f6587b23434d52e13c167fc0044293b128ff03cc2e47b23cc2545e8a67 WatchSource:0}: Error finding container e5e105f6587b23434d52e13c167fc0044293b128ff03cc2e47b23cc2545e8a67: Status 404 returned error can't find the container with id e5e105f6587b23434d52e13c167fc0044293b128ff03cc2e47b23cc2545e8a67 Jul 14 19:48:36.903964 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:36.903918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rvfr" event={"ID":"daa50ea0-e19e-4c48-9afc-adc079abf2d5","Type":"ContainerStarted","Data":"e5e105f6587b23434d52e13c167fc0044293b128ff03cc2e47b23cc2545e8a67"} Jul 14 19:48:36.905870 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:36.905845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvd8p" event={"ID":"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f","Type":"ContainerStarted","Data":"99aa4c5c5f0924e7e259480b25b380ea9083bbad98c957d4dcaf4ce1bfe0cda5"} Jul 14 19:48:41.933930 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.933887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c8w" event={"ID":"99b20e35-bc5e-44f8-94de-2828be472ec0","Type":"ContainerStarted","Data":"f0ca9f78090b3e766b8f20acea33eef141e314890584c7608c0da9dfeff60bab"} Jul 14 19:48:41.934400 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.933985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s5c8w" event={"ID":"99b20e35-bc5e-44f8-94de-2828be472ec0","Type":"ContainerStarted","Data":"8d74ff50db4d9ce0aff036cd7696e44d705d44b9f95bd5245182084393b3c57e"} Jul 14 19:48:41.935441 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.935416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8rvfr" event={"ID":"daa50ea0-e19e-4c48-9afc-adc079abf2d5","Type":"ContainerStarted","Data":"f4674dabe7037860f1529f7a6f49af3c9fb8a74bddf71a76ea149137c36fc151"} Jul 14 19:48:41.936867 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.936843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zvd8p" event={"ID":"d58b8886-eeeb-4add-bde3-c9b3a4bc2d7f","Type":"ContainerStarted","Data":"841cc2de4c705868401c3d80c376fae3fb23abe14f84422f740c2d5020f01948"} Jul 14 19:48:41.936997 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.936982 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:48:41.952467 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.952421 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s5c8w" podStartSLOduration=66.970584685 podStartE2EDuration="1m11.952409437s" totalImagesPullingTime="4.981824752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.950926307 +0000 UTC m=+71.795263182" watchObservedRunningTime="2026-07-14 19:48:41.952409437 +0000 UTC m=+71.796746312" Jul 14 19:48:41.967693 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.967652 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8rvfr" podStartSLOduration=67.193599377 podStartE2EDuration="1m11.967639168s" totalImagesPullingTime="4.774039791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.966911536 +0000 UTC m=+71.811248425" watchObservedRunningTime="2026-07-14 19:48:41.967639168 +0000 UTC m=+71.811976042" Jul 14 19:48:41.981949 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:41.981912 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zvd8p" podStartSLOduration=67.224470826 podStartE2EDuration="1m11.981902214s" totalImagesPullingTime="4.757431388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:48:41.98116741 +0000 UTC m=+71.825504284" watchObservedRunningTime="2026-07-14 19:48:41.981902214 +0000 UTC m=+71.826239087" Jul 14 19:48:52.508735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:52.508626 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:48:52.508735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:48:52.508712 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:49:12.514181 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:12.514151 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:49:12.518088 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:12.518067 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b55f44bb6-fqpkh" Jul 14 19:49:12.944383 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:12.944356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zvd8p" Jul 14 19:49:52.262158 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.262121 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b"] Jul 14 19:49:52.265627 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.265610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.269407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.268765 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 19:49:52.269407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.268986 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 19:49:52.269407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.269147 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tz9k6\"" Jul 14 19:49:52.269407 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.269346 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 19:49:52.269730 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.269516 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 19:49:52.269730 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.269645 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 19:49:52.274738 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.274706 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 19:49:52.276749 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.276724 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b"] Jul 14 19:49:52.419934 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.419898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.419934 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.419934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-serving-certs-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.419956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.420033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.420065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.420089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-federate-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.420117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhwx\" (UniqueName: \"kubernetes.io/projected/1d09d6b1-222c-43a2-b4c8-22c49e746e71-kube-api-access-nxhwx\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.420284 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.420167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-metrics-client-ca\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521475 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521475 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-serving-certs-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521475 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-federate-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhwx\" (UniqueName: \"kubernetes.io/projected/1d09d6b1-222c-43a2-b4c8-22c49e746e71-kube-api-access-nxhwx\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.521778 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.521617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-metrics-client-ca\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.522194 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.522164 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-serving-certs-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.522381 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.522347 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-metrics-client-ca\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.523145 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.523118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.524068 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.524043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-telemeter-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.524245 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.524225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.524460 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.524440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-federate-client-tls\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.524543 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.524440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d09d6b1-222c-43a2-b4c8-22c49e746e71-secret-telemeter-client\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.530836 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.530811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhwx\" (UniqueName: \"kubernetes.io/projected/1d09d6b1-222c-43a2-b4c8-22c49e746e71-kube-api-access-nxhwx\") pod \"telemeter-client-7bd7f5567c-k6x2b\" (UID: \"1d09d6b1-222c-43a2-b4c8-22c49e746e71\") " pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.577603 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.577584 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" Jul 14 19:49:52.715549 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:52.715462 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b"] Jul 14 19:49:52.717685 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:49:52.717650 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d09d6b1_222c_43a2_b4c8_22c49e746e71.slice/crio-12075c4c688bf0aff44c4940243ef81a9ffced81b47cabec2307630f0d4355c6 WatchSource:0}: Error finding container 12075c4c688bf0aff44c4940243ef81a9ffced81b47cabec2307630f0d4355c6: Status 404 returned error can't find the container with id 12075c4c688bf0aff44c4940243ef81a9ffced81b47cabec2307630f0d4355c6 Jul 14 19:49:53.177584 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:53.177544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" event={"ID":"1d09d6b1-222c-43a2-b4c8-22c49e746e71","Type":"ContainerStarted","Data":"12075c4c688bf0aff44c4940243ef81a9ffced81b47cabec2307630f0d4355c6"} Jul 14 19:49:55.186682 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:55.186639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" event={"ID":"1d09d6b1-222c-43a2-b4c8-22c49e746e71","Type":"ContainerStarted","Data":"acc5a18b84368dc2dbe14ccd88a4c0a610c3f75dabf0c01f52433ee306796aaa"} Jul 14 19:49:56.192256 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:56.192228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" event={"ID":"1d09d6b1-222c-43a2-b4c8-22c49e746e71","Type":"ContainerStarted","Data":"3d8c7f9af2682be453a08adb78394ffaa07b2d82947dcf7626f0641c96aa38c7"} Jul 14 19:49:57.197205 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:57.197165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" event={"ID":"1d09d6b1-222c-43a2-b4c8-22c49e746e71","Type":"ContainerStarted","Data":"9d269ab8017f8841d1004ccc93af3369aecf444db1301733aef414953bdd3437"} Jul 14 19:49:57.220379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:49:57.220327 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7bd7f5567c-k6x2b" podStartSLOduration=1.92907559 podStartE2EDuration="5.220310279s" totalImagesPullingTime="3.291234689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:49:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:49:57.218152232 +0000 UTC m=+147.062489107" watchObservedRunningTime="2026-07-14 19:49:57.220310279 +0000 UTC m=+147.064647152" Jul 14 19:51:07.599977 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.599949 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt"] Jul 14 19:51:07.605401 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.604195 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.607483 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.607463 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:51:07.608264 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.608249 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:51:07.608320 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.608249 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhxbs\"" Jul 14 19:51:07.614319 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.614287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt"] Jul 14 19:51:07.765838 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.765806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.765991 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.765859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.765991 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.765890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.866809 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.866736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.866809 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.866780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.866999 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.866827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.867127 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.867106 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.867202 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.867170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.882315 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.882292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:07.913630 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:07.913605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:08.042047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:08.042022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt"] Jul 14 19:51:08.044622 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:08.044595 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9db37f_de9f_47db_803e_d2e170800e77.slice/crio-ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f WatchSource:0}: Error finding container ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f: Status 404 returned error can't find the container with id ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f Jul 14 19:51:08.429581 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:08.429549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" event={"ID":"8c9db37f-de9f-47db-803e-d2e170800e77","Type":"ContainerStarted","Data":"ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f"} Jul 14 19:51:14.455173 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:14.455133 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c9db37f-de9f-47db-803e-d2e170800e77" containerID="458e6fbd5dd40f3ff727481d92a559ec0d317458383bf2102601c9bd94923051" exitCode=0 Jul 14 19:51:14.455564 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:14.455213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" event={"ID":"8c9db37f-de9f-47db-803e-d2e170800e77","Type":"ContainerDied","Data":"458e6fbd5dd40f3ff727481d92a559ec0d317458383bf2102601c9bd94923051"} Jul 14 19:51:16.464742 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:16.464709 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c9db37f-de9f-47db-803e-d2e170800e77" containerID="3dd594bec86f5b1707d28d4102e5c8b8277b7fb63583f46ece7d39e0d596f181" exitCode=0 Jul 14 19:51:16.465032 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:16.464759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" event={"ID":"8c9db37f-de9f-47db-803e-d2e170800e77","Type":"ContainerDied","Data":"3dd594bec86f5b1707d28d4102e5c8b8277b7fb63583f46ece7d39e0d596f181"} Jul 14 19:51:26.501220 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:26.501186 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c9db37f-de9f-47db-803e-d2e170800e77" containerID="0038e4b56b3082a7d3a5f0426554d89db8689b314c9b33a784a94efd0ddc634d" exitCode=0 Jul 14 19:51:26.501220 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:26.501225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" event={"ID":"8c9db37f-de9f-47db-803e-d2e170800e77","Type":"ContainerDied","Data":"0038e4b56b3082a7d3a5f0426554d89db8689b314c9b33a784a94efd0ddc634d"} Jul 14 19:51:27.635042 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.635020 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:27.725181 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.725151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db\") pod \"8c9db37f-de9f-47db-803e-d2e170800e77\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " Jul 14 19:51:27.725363 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.725194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle\") pod \"8c9db37f-de9f-47db-803e-d2e170800e77\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " Jul 14 19:51:27.725363 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.725224 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util\") pod \"8c9db37f-de9f-47db-803e-d2e170800e77\" (UID: \"8c9db37f-de9f-47db-803e-d2e170800e77\") " Jul 14 19:51:27.725937 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.725907 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle" (OuterVolumeSpecName: "bundle") pod "8c9db37f-de9f-47db-803e-d2e170800e77" (UID: "8c9db37f-de9f-47db-803e-d2e170800e77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:27.727625 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.727601 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db" (OuterVolumeSpecName: "kube-api-access-mk6db") pod "8c9db37f-de9f-47db-803e-d2e170800e77" (UID: "8c9db37f-de9f-47db-803e-d2e170800e77"). InnerVolumeSpecName "kube-api-access-mk6db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:27.729562 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.729540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util" (OuterVolumeSpecName: "util") pod "8c9db37f-de9f-47db-803e-d2e170800e77" (UID: "8c9db37f-de9f-47db-803e-d2e170800e77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:27.826042 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.825957 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk6db\" (UniqueName: \"kubernetes.io/projected/8c9db37f-de9f-47db-803e-d2e170800e77-kube-api-access-mk6db\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:27.826042 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.825997 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:27.826042 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:27.826007 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c9db37f-de9f-47db-803e-d2e170800e77-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:28.510479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:28.510452 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" Jul 14 19:51:28.510635 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:28.510448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zd7mt" event={"ID":"8c9db37f-de9f-47db-803e-d2e170800e77","Type":"ContainerDied","Data":"ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f"} Jul 14 19:51:28.510635 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:28.510546 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad800b3b54a2d73b54267e415d8ed882e10054f4b7a30e5254ce95586e73887f" Jul 14 19:51:35.370450 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370415 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf"] Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370705 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="util" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370717 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="util" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370727 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="extract" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370732 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="extract" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370758 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="pull" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370764 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="pull" Jul 14 19:51:35.370987 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.370830 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c9db37f-de9f-47db-803e-d2e170800e77" containerName="extract" Jul 14 19:51:35.376814 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.376794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.379542 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.379522 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 19:51:35.379542 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.379525 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-crgtl\"" Jul 14 19:51:35.379718 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.379550 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:51:35.391298 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.391275 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf"] Jul 14 19:51:35.477106 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.477081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rt7h\" (UniqueName: \"kubernetes.io/projected/f7117e5c-7c60-4bf7-b6e8-2029d083218a-kube-api-access-8rt7h\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.477228 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.477127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7117e5c-7c60-4bf7-b6e8-2029d083218a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.578019 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.577991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rt7h\" (UniqueName: \"kubernetes.io/projected/f7117e5c-7c60-4bf7-b6e8-2029d083218a-kube-api-access-8rt7h\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.578157 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.578038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7117e5c-7c60-4bf7-b6e8-2029d083218a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.578356 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.578341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7117e5c-7c60-4bf7-b6e8-2029d083218a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.587177 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.587148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rt7h\" (UniqueName: \"kubernetes.io/projected/f7117e5c-7c60-4bf7-b6e8-2029d083218a-kube-api-access-8rt7h\") pod \"cert-manager-operator-controller-manager-55c444bf86-d2rcf\" (UID: \"f7117e5c-7c60-4bf7-b6e8-2029d083218a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.685614 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.685539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" Jul 14 19:51:35.814861 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:35.814831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf"] Jul 14 19:51:35.818449 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:35.818418 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7117e5c_7c60_4bf7_b6e8_2029d083218a.slice/crio-a97d0ecfc091ea45e6f52c3827a177c97af2247884514f2ed97914c81bbd3f32 WatchSource:0}: Error finding container a97d0ecfc091ea45e6f52c3827a177c97af2247884514f2ed97914c81bbd3f32: Status 404 returned error can't find the container with id a97d0ecfc091ea45e6f52c3827a177c97af2247884514f2ed97914c81bbd3f32 Jul 14 19:51:36.546101 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:36.546065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" event={"ID":"f7117e5c-7c60-4bf7-b6e8-2029d083218a","Type":"ContainerStarted","Data":"a97d0ecfc091ea45e6f52c3827a177c97af2247884514f2ed97914c81bbd3f32"} Jul 14 19:51:37.551637 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:37.551601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" event={"ID":"f7117e5c-7c60-4bf7-b6e8-2029d083218a","Type":"ContainerStarted","Data":"3364dfe34c0cad66e6b886813020c3ad766ba5319e3bc039624843fc20c12354"} Jul 14 19:51:37.576471 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:37.576382 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-d2rcf" podStartSLOduration=1.07602507 podStartE2EDuration="2.576369266s" totalImagesPullingTime="1.500344196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:37.574112433 +0000 UTC m=+247.418449307" watchObservedRunningTime="2026-07-14 19:51:37.576369266 +0000 UTC m=+247.420706139" Jul 14 19:51:38.872677 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.872641 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp"] Jul 14 19:51:38.875993 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.875977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:38.878431 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.878410 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:51:38.878595 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.878476 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:51:38.879377 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.879363 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhxbs\"" Jul 14 19:51:38.887355 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.887337 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp"] Jul 14 19:51:38.903350 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.903325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:38.903465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.903383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxff\" (UniqueName: \"kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:38.903559 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:38.903511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.004857 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.004831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.004963 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.004875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.004963 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.004909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxff\" (UniqueName: \"kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.005184 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.005166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.005267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.005249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.014398 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.014372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxff\" (UniqueName: \"kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.185077 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.184980 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:39.322157 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.321983 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp"] Jul 14 19:51:39.324601 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:39.324560 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67d1c86_d966_4ae1_b23a_59715adcbbc0.slice/crio-e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b WatchSource:0}: Error finding container e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b: Status 404 returned error can't find the container with id e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b Jul 14 19:51:39.560338 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.560303 2574 generic.go:356] "Generic (PLEG): container finished" podID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerID="993d53ff1d63cd89292a7be67670c6793a250a55a082c205b184bed76d8ee2d5" exitCode=0 Jul 14 19:51:39.560556 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.560373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" event={"ID":"b67d1c86-d966-4ae1-b23a-59715adcbbc0","Type":"ContainerDied","Data":"993d53ff1d63cd89292a7be67670c6793a250a55a082c205b184bed76d8ee2d5"} Jul 14 19:51:39.560556 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:39.560411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" event={"ID":"b67d1c86-d966-4ae1-b23a-59715adcbbc0","Type":"ContainerStarted","Data":"e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b"} Jul 14 19:51:40.626014 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.625980 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhz55"] Jul 14 19:51:40.629038 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.629015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.631765 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.631742 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 19:51:40.632771 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.632745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-h6b24\"" Jul 14 19:51:40.632771 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.632752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 19:51:40.640473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.640444 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhz55"] Jul 14 19:51:40.718142 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.718114 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.718282 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.718161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpwx\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-kube-api-access-9lpwx\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.818577 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.818545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.818753 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.818604 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpwx\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-kube-api-access-9lpwx\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.828369 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.828345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpwx\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-kube-api-access-9lpwx\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.828510 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.828452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25e09648-6c47-4fed-80d1-a4d9e84bfb5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zhz55\" (UID: \"25e09648-6c47-4fed-80d1-a4d9e84bfb5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:40.946465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:40.946439 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" Jul 14 19:51:41.770616 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:41.770590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zhz55"] Jul 14 19:51:41.773700 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:41.773677 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25e09648_6c47_4fed_80d1_a4d9e84bfb5f.slice/crio-517f95fd9b8591cd643763aa127e8d776dccb2c59a9b5055db984b4f6d32c233 WatchSource:0}: Error finding container 517f95fd9b8591cd643763aa127e8d776dccb2c59a9b5055db984b4f6d32c233: Status 404 returned error can't find the container with id 517f95fd9b8591cd643763aa127e8d776dccb2c59a9b5055db984b4f6d32c233 Jul 14 19:51:42.579437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:42.579393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" event={"ID":"25e09648-6c47-4fed-80d1-a4d9e84bfb5f","Type":"ContainerStarted","Data":"517f95fd9b8591cd643763aa127e8d776dccb2c59a9b5055db984b4f6d32c233"} Jul 14 19:51:42.581440 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:42.581413 2574 generic.go:356] "Generic (PLEG): container finished" podID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerID="1925faaf59ead3dd2f1f4ada9b110f82d823864e054940166e8fb9e1250d61f9" exitCode=0 Jul 14 19:51:42.581577 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:42.581467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" event={"ID":"b67d1c86-d966-4ae1-b23a-59715adcbbc0","Type":"ContainerDied","Data":"1925faaf59ead3dd2f1f4ada9b110f82d823864e054940166e8fb9e1250d61f9"} Jul 14 19:51:43.588802 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:43.588768 2574 generic.go:356] "Generic (PLEG): container finished" podID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerID="44dfbf1ea575189c269c20b4d619f6dbd4ed8397d559b9e32ee34acf94c7f135" exitCode=0 Jul 14 19:51:43.589191 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:43.588833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" event={"ID":"b67d1c86-d966-4ae1-b23a-59715adcbbc0","Type":"ContainerDied","Data":"44dfbf1ea575189c269c20b4d619f6dbd4ed8397d559b9e32ee34acf94c7f135"} Jul 14 19:51:44.599425 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.599357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" event={"ID":"25e09648-6c47-4fed-80d1-a4d9e84bfb5f","Type":"ContainerStarted","Data":"7b7f76b1f48f7ad9f1b9bddb4f8f886bca3922294e6a89409558197af41ec6f1"} Jul 14 19:51:44.621529 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.621442 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zhz55" podStartSLOduration=2.11991455 podStartE2EDuration="4.621418643s" totalImagesPullingTime="2.501504093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:44.618950095 +0000 UTC m=+254.463286970" watchObservedRunningTime="2026-07-14 19:51:44.621418643 +0000 UTC m=+254.465755520" Jul 14 19:51:44.733447 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.733420 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:44.850410 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.850181 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util\") pod \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " Jul 14 19:51:44.850410 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.850245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhxff\" (UniqueName: \"kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff\") pod \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " Jul 14 19:51:44.850410 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.850269 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle\") pod \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\" (UID: \"b67d1c86-d966-4ae1-b23a-59715adcbbc0\") " Jul 14 19:51:44.850726 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.850671 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle" (OuterVolumeSpecName: "bundle") pod "b67d1c86-d966-4ae1-b23a-59715adcbbc0" (UID: "b67d1c86-d966-4ae1-b23a-59715adcbbc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:44.852466 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.852431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff" (OuterVolumeSpecName: "kube-api-access-qhxff") pod "b67d1c86-d966-4ae1-b23a-59715adcbbc0" (UID: "b67d1c86-d966-4ae1-b23a-59715adcbbc0"). InnerVolumeSpecName "kube-api-access-qhxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:51:44.856034 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.856006 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util" (OuterVolumeSpecName: "util") pod "b67d1c86-d966-4ae1-b23a-59715adcbbc0" (UID: "b67d1c86-d966-4ae1-b23a-59715adcbbc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:51:44.951437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.951400 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:44.951437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.951431 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhxff\" (UniqueName: \"kubernetes.io/projected/b67d1c86-d966-4ae1-b23a-59715adcbbc0-kube-api-access-qhxff\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:44.951437 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:44.951441 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b67d1c86-d966-4ae1-b23a-59715adcbbc0-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:51:45.483164 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483134 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f5fp4"] Jul 14 19:51:45.483432 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483420 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="util" Jul 14 19:51:45.483511 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483435 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="util" Jul 14 19:51:45.483511 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483451 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="pull" Jul 14 19:51:45.483511 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483457 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="pull" Jul 14 19:51:45.483511 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483464 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="extract" Jul 14 19:51:45.483511 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483469 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="extract" Jul 14 19:51:45.483654 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.483541 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b67d1c86-d966-4ae1-b23a-59715adcbbc0" containerName="extract" Jul 14 19:51:45.486508 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.486479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.490238 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.490217 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-svfdg\"" Jul 14 19:51:45.496589 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.496551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f5fp4"] Jul 14 19:51:45.556473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.556442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmc9\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-kube-api-access-tdmc9\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.556633 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.556485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.600200 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.600176 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" Jul 14 19:51:45.600615 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.600210 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv7rfp" event={"ID":"b67d1c86-d966-4ae1-b23a-59715adcbbc0","Type":"ContainerDied","Data":"e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b"} Jul 14 19:51:45.600615 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.600260 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e982d507a6e4bb1adc17a8a2ccf479ae8bb5c2961dd1d0eb96060edaf183fd5b" Jul 14 19:51:45.657283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.657259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmc9\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-kube-api-access-tdmc9\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.657420 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.657295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.667703 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.667674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.667800 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.667758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmc9\" (UniqueName: \"kubernetes.io/projected/eac2518c-186e-404a-a7fc-8c4c4d3a2ba6-kube-api-access-tdmc9\") pod \"cert-manager-webhook-6486ffd997-f5fp4\" (UID: \"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.795203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.795124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:45.920238 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:45.920209 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f5fp4"] Jul 14 19:51:45.922467 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:45.922440 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeac2518c_186e_404a_a7fc_8c4c4d3a2ba6.slice/crio-ade963930ecc1798f37cb44a9687b4dcfab2b97aa8d71ea1d8a57a34f82d5f45 WatchSource:0}: Error finding container ade963930ecc1798f37cb44a9687b4dcfab2b97aa8d71ea1d8a57a34f82d5f45: Status 404 returned error can't find the container with id ade963930ecc1798f37cb44a9687b4dcfab2b97aa8d71ea1d8a57a34f82d5f45 Jul 14 19:51:46.605119 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:46.605046 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" event={"ID":"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6","Type":"ContainerStarted","Data":"a33f15b8f48c50bf35f818818494e7ec6d872a806c659315e3ac9e3004381491"} Jul 14 19:51:46.605119 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:46.605121 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:51:46.605553 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:46.605133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" event={"ID":"eac2518c-186e-404a-a7fc-8c4c4d3a2ba6","Type":"ContainerStarted","Data":"ade963930ecc1798f37cb44a9687b4dcfab2b97aa8d71ea1d8a57a34f82d5f45"} Jul 14 19:51:46.622409 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:46.622361 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" podStartSLOduration=1.622346189 podStartE2EDuration="1.622346189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:46.62062728 +0000 UTC m=+256.464964160" watchObservedRunningTime="2026-07-14 19:51:46.622346189 +0000 UTC m=+256.466683062" Jul 14 19:51:49.711668 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.711630 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2"] Jul 14 19:51:49.714735 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.714715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.717196 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.717171 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:51:49.717309 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.717250 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kbtg9\"" Jul 14 19:51:49.718267 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.718244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 19:51:49.723397 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.723377 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2"] Jul 14 19:51:49.790612 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.790582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdab5eea-37f6-4969-8919-89db79a70d03-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.790739 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.790630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82p2w\" (UniqueName: \"kubernetes.io/projected/cdab5eea-37f6-4969-8919-89db79a70d03-kube-api-access-82p2w\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.891697 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.891670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82p2w\" (UniqueName: \"kubernetes.io/projected/cdab5eea-37f6-4969-8919-89db79a70d03-kube-api-access-82p2w\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.891799 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.891731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdab5eea-37f6-4969-8919-89db79a70d03-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.892043 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.892027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cdab5eea-37f6-4969-8919-89db79a70d03-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:49.900607 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:49.900578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82p2w\" (UniqueName: \"kubernetes.io/projected/cdab5eea-37f6-4969-8919-89db79a70d03-kube-api-access-82p2w\") pod \"openshift-lws-operator-fd8ccff4c-wbjc2\" (UID: \"cdab5eea-37f6-4969-8919-89db79a70d03\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:50.025384 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:50.025327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" Jul 14 19:51:50.152888 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:50.152863 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2"] Jul 14 19:51:50.155254 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:51:50.155228 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdab5eea_37f6_4969_8919_89db79a70d03.slice/crio-3f6fd66dbb39aaa49f7962e37222524d30795a083d04a55258a05e9a1bccc5d5 WatchSource:0}: Error finding container 3f6fd66dbb39aaa49f7962e37222524d30795a083d04a55258a05e9a1bccc5d5: Status 404 returned error can't find the container with id 3f6fd66dbb39aaa49f7962e37222524d30795a083d04a55258a05e9a1bccc5d5 Jul 14 19:51:50.622393 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:50.622360 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" event={"ID":"cdab5eea-37f6-4969-8919-89db79a70d03","Type":"ContainerStarted","Data":"3f6fd66dbb39aaa49f7962e37222524d30795a083d04a55258a05e9a1bccc5d5"} Jul 14 19:51:52.613265 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:51:52.613199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f5fp4" Jul 14 19:52:02.673839 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:02.673799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" event={"ID":"cdab5eea-37f6-4969-8919-89db79a70d03","Type":"ContainerStarted","Data":"7aea4a5465db6904f76ee9d39330bfc0eccf9afec8121a26ef801f2b8f7c5a83"} Jul 14 19:52:02.693246 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:02.693196 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wbjc2" podStartSLOduration=2.118503308 podStartE2EDuration="13.693181066s" totalImagesPullingTime="11.574677758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:02.691523032 +0000 UTC m=+272.535859907" watchObservedRunningTime="2026-07-14 19:52:02.693181066 +0000 UTC m=+272.537517939" Jul 14 19:52:04.904783 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.904749 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd"] Jul 14 19:52:04.906940 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.906922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:04.909318 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.909296 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhxbs\"" Jul 14 19:52:04.909429 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.909341 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 19:52:04.910326 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.910305 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 19:52:04.916092 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:04.916073 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd"] Jul 14 19:52:05.007328 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.007298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2l9\" (UniqueName: \"kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.007458 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.007388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.007458 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.007424 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.108387 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.108349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.108546 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.108398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.108546 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.108444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2l9\" (UniqueName: \"kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.108757 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.108739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.108825 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.108805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.117619 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.117597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2l9\" (UniqueName: \"kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.216771 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.216742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:05.345082 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.345060 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd"] Jul 14 19:52:05.347701 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:52:05.347675 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc6e80d_aade_47cf_a1cb_26093c1ed2b6.slice/crio-7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d WatchSource:0}: Error finding container 7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d: Status 404 returned error can't find the container with id 7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d Jul 14 19:52:05.687922 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.687833 2574 generic.go:356] "Generic (PLEG): container finished" podID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerID="54bb0ce2ce38dbb7ef9ba41596db5ba45ca783e36ecb1f699106f91b15f11773" exitCode=0 Jul 14 19:52:05.688049 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.687915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" event={"ID":"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6","Type":"ContainerDied","Data":"54bb0ce2ce38dbb7ef9ba41596db5ba45ca783e36ecb1f699106f91b15f11773"} Jul 14 19:52:05.688049 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:05.687968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" event={"ID":"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6","Type":"ContainerStarted","Data":"7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d"} Jul 14 19:52:06.693859 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:06.693826 2574 generic.go:356] "Generic (PLEG): container finished" podID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerID="3f917e76dd137d358d727a9226bf29cab79f33f350c7603029883637513f222d" exitCode=0 Jul 14 19:52:06.694280 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:06.693906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" event={"ID":"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6","Type":"ContainerDied","Data":"3f917e76dd137d358d727a9226bf29cab79f33f350c7603029883637513f222d"} Jul 14 19:52:07.700206 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:07.700171 2574 generic.go:356] "Generic (PLEG): container finished" podID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerID="861d20fff5ecf7bb1975eb0b3c2600532c72e61f8af1561e4a46ee5503e8b667" exitCode=0 Jul 14 19:52:07.700604 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:07.700258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" event={"ID":"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6","Type":"ContainerDied","Data":"861d20fff5ecf7bb1975eb0b3c2600532c72e61f8af1561e4a46ee5503e8b667"} Jul 14 19:52:08.837112 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.837089 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:08.937686 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.937663 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle\") pod \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " Jul 14 19:52:08.937841 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.937745 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr2l9\" (UniqueName: \"kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9\") pod \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " Jul 14 19:52:08.937841 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.937767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util\") pod \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\" (UID: \"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6\") " Jul 14 19:52:08.938536 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.938482 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle" (OuterVolumeSpecName: "bundle") pod "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" (UID: "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:08.939861 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.939840 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9" (OuterVolumeSpecName: "kube-api-access-dr2l9") pod "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" (UID: "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6"). InnerVolumeSpecName "kube-api-access-dr2l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:52:08.943525 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:08.943478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util" (OuterVolumeSpecName: "util") pod "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" (UID: "6cc6e80d-aade-47cf-a1cb-26093c1ed2b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:52:09.038849 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.038782 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr2l9\" (UniqueName: \"kubernetes.io/projected/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-kube-api-access-dr2l9\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:52:09.038849 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.038808 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:52:09.038849 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.038817 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc6e80d-aade-47cf-a1cb-26093c1ed2b6-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:52:09.710056 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.710020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" event={"ID":"6cc6e80d-aade-47cf-a1cb-26093c1ed2b6","Type":"ContainerDied","Data":"7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d"} Jul 14 19:52:09.710056 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.710054 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs54zd" Jul 14 19:52:09.710248 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:09.710076 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8c64da0eec7afd54a023d7ce6f6b68a764026adf861c2d582b7257c837fb1d" Jul 14 19:52:19.342598 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342563 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj"] Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342866 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="pull" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342879 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="pull" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342897 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="extract" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342902 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="extract" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342910 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="util" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342915 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="util" Jul 14 19:52:19.343001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.342970 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc6e80d-aade-47cf-a1cb-26093c1ed2b6" containerName="extract" Jul 14 19:52:19.348897 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.348878 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.351932 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.351907 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:52:19.352053 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.351985 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:52:19.352119 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.352102 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mfbvv\"" Jul 14 19:52:19.352524 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.352474 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:52:19.352820 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.352794 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:52:19.362138 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.362118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj"] Jul 14 19:52:19.418027 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.417996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.418177 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.418034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.418177 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.418074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.418289 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.418175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghl4d\" (UniqueName: \"kubernetes.io/projected/0894dd72-2728-437f-a9de-eb6d75738a92-kube-api-access-ghl4d\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.418289 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.418225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519122 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghl4d\" (UniqueName: \"kubernetes.io/projected/0894dd72-2728-437f-a9de-eb6d75738a92-kube-api-access-ghl4d\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519272 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519272 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519272 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519272 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.519655 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.519630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-manifests\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.521613 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.521588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0894dd72-2728-437f-a9de-eb6d75738a92-tmp\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.521882 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.521860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.521964 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.521896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0894dd72-2728-437f-a9de-eb6d75738a92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.530945 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.530915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghl4d\" (UniqueName: \"kubernetes.io/projected/0894dd72-2728-437f-a9de-eb6d75738a92-kube-api-access-ghl4d\") pod \"opendatahub-operator-controller-manager-6c597f769-gk7lj\" (UID: \"0894dd72-2728-437f-a9de-eb6d75738a92\") " pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.659762 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.659680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:19.795473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:19.795445 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj"] Jul 14 19:52:19.797325 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:52:19.797302 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0894dd72_2728_437f_a9de_eb6d75738a92.slice/crio-419f6b3367031c056e246a7ca7a102d27363b06d6dfcd18e68a9f36a7d608816 WatchSource:0}: Error finding container 419f6b3367031c056e246a7ca7a102d27363b06d6dfcd18e68a9f36a7d608816: Status 404 returned error can't find the container with id 419f6b3367031c056e246a7ca7a102d27363b06d6dfcd18e68a9f36a7d608816 Jul 14 19:52:20.762825 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:20.762784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" event={"ID":"0894dd72-2728-437f-a9de-eb6d75738a92","Type":"ContainerStarted","Data":"419f6b3367031c056e246a7ca7a102d27363b06d6dfcd18e68a9f36a7d608816"} Jul 14 19:52:23.778700 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:23.778662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" event={"ID":"0894dd72-2728-437f-a9de-eb6d75738a92","Type":"ContainerStarted","Data":"b4919936e5bb27eca618008a6a44d15d34b963c83063ee72d55fdc402a14ef3f"} Jul 14 19:52:24.784506 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:24.784450 2574 generic.go:356] "Generic (PLEG): container finished" podID="0894dd72-2728-437f-a9de-eb6d75738a92" containerID="b4919936e5bb27eca618008a6a44d15d34b963c83063ee72d55fdc402a14ef3f" exitCode=0 Jul 14 19:52:24.784884 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:24.784539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" event={"ID":"0894dd72-2728-437f-a9de-eb6d75738a92","Type":"ContainerDied","Data":"b4919936e5bb27eca618008a6a44d15d34b963c83063ee72d55fdc402a14ef3f"} Jul 14 19:52:25.790882 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:25.790847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" event={"ID":"0894dd72-2728-437f-a9de-eb6d75738a92","Type":"ContainerStarted","Data":"715477deb3eabd59233aaef6169ed25e47e540deecaea64ac019db781a2492f4"} Jul 14 19:52:25.791281 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:25.790971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:25.813700 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:25.813655 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" podStartSLOduration=2.224833669 podStartE2EDuration="6.813641965s" totalImagesPullingTime="3.570326154s" totalInitContainerRuntime="1.018482142s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:25.810751112 +0000 UTC m=+295.655088009" watchObservedRunningTime="2026-07-14 19:52:25.813641965 +0000 UTC m=+295.657978839" Jul 14 19:52:30.498302 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:30.498272 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:52:36.796960 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:36.796930 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c597f769-gk7lj" Jul 14 19:52:39.064656 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.064620 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5b6jd"] Jul 14 19:52:39.068509 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.068467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.072630 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.072608 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 19:52:39.072761 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.072669 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8wbn7\"" Jul 14 19:52:39.081599 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.081575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5b6jd"] Jul 14 19:52:39.182094 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.182061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xn7f\" (UniqueName: \"kubernetes.io/projected/07ae2cb9-a573-41ef-b100-3f643925c6c4-kube-api-access-2xn7f\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.182257 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.182147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07ae2cb9-a573-41ef-b100-3f643925c6c4-cert\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.283508 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.283455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07ae2cb9-a573-41ef-b100-3f643925c6c4-cert\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.283670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.283542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xn7f\" (UniqueName: \"kubernetes.io/projected/07ae2cb9-a573-41ef-b100-3f643925c6c4-kube-api-access-2xn7f\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.286036 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.286013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07ae2cb9-a573-41ef-b100-3f643925c6c4-cert\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.297092 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.297068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xn7f\" (UniqueName: \"kubernetes.io/projected/07ae2cb9-a573-41ef-b100-3f643925c6c4-kube-api-access-2xn7f\") pod \"odh-model-controller-7998c67dc4-5b6jd\" (UID: \"07ae2cb9-a573-41ef-b100-3f643925c6c4\") " pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.379479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.379408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:39.517088 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.517060 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-5b6jd"] Jul 14 19:52:39.519549 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:52:39.519521 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ae2cb9_a573_41ef_b100_3f643925c6c4.slice/crio-a0cde45e3fc84e4f6b3eca0894c3ca8c087f687f57ea1586e65e99be16c1f41d WatchSource:0}: Error finding container a0cde45e3fc84e4f6b3eca0894c3ca8c087f687f57ea1586e65e99be16c1f41d: Status 404 returned error can't find the container with id a0cde45e3fc84e4f6b3eca0894c3ca8c087f687f57ea1586e65e99be16c1f41d Jul 14 19:52:39.521060 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.521041 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:52:39.849563 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:39.849528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" event={"ID":"07ae2cb9-a573-41ef-b100-3f643925c6c4","Type":"ContainerStarted","Data":"a0cde45e3fc84e4f6b3eca0894c3ca8c087f687f57ea1586e65e99be16c1f41d"} Jul 14 19:52:43.869297 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:43.869209 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" event={"ID":"07ae2cb9-a573-41ef-b100-3f643925c6c4","Type":"ContainerStarted","Data":"2608d83b8029969fc55b4decb5750146d0972720fdaf9c2e9168d1b9a61f7497"} Jul 14 19:52:43.869694 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:43.869335 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:43.886083 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:43.886039 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" podStartSLOduration=0.870534021 podStartE2EDuration="4.886026495s" totalImagesPullingTime="4.015492474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:43.885406115 +0000 UTC m=+313.729742989" watchObservedRunningTime="2026-07-14 19:52:43.886026495 +0000 UTC m=+313.730363369" Jul 14 19:52:46.609484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.609448 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx"] Jul 14 19:52:46.612929 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.612908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.615744 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.615722 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 19:52:46.617036 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.617014 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 19:52:46.617128 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.617015 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kvspf\"" Jul 14 19:52:46.617128 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.617013 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 19:52:46.622386 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.622335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx"] Jul 14 19:52:46.636760 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.636738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.636881 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.636782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.636881 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.636839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44a10a91-5fe6-4150-9567-874fdf632116-manager-config\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.636991 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.636915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccc8\" (UniqueName: \"kubernetes.io/projected/44a10a91-5fe6-4150-9567-874fdf632116-kube-api-access-sccc8\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.738039 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.738004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccc8\" (UniqueName: \"kubernetes.io/projected/44a10a91-5fe6-4150-9567-874fdf632116-kube-api-access-sccc8\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.738039 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.738041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.738213 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.738169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.738257 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.738218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44a10a91-5fe6-4150-9567-874fdf632116-manager-config\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.738879 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.738860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44a10a91-5fe6-4150-9567-874fdf632116-manager-config\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.740594 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.740575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-metrics-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.740763 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.740744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44a10a91-5fe6-4150-9567-874fdf632116-cert\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.748116 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.748095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccc8\" (UniqueName: \"kubernetes.io/projected/44a10a91-5fe6-4150-9567-874fdf632116-kube-api-access-sccc8\") pod \"lws-controller-manager-77d6d59f96-62vfx\" (UID: \"44a10a91-5fe6-4150-9567-874fdf632116\") " pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:46.924414 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:46.924349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:47.052345 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:47.052322 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx"] Jul 14 19:52:47.054693 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:52:47.054670 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a10a91_5fe6_4150_9567_874fdf632116.slice/crio-6c1076c65766a2f49eaad07c1428f9e2460bae6345237b7db1d1ecd439ba0d27 WatchSource:0}: Error finding container 6c1076c65766a2f49eaad07c1428f9e2460bae6345237b7db1d1ecd439ba0d27: Status 404 returned error can't find the container with id 6c1076c65766a2f49eaad07c1428f9e2460bae6345237b7db1d1ecd439ba0d27 Jul 14 19:52:47.889048 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:47.889009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" event={"ID":"44a10a91-5fe6-4150-9567-874fdf632116","Type":"ContainerStarted","Data":"6c1076c65766a2f49eaad07c1428f9e2460bae6345237b7db1d1ecd439ba0d27"} Jul 14 19:52:48.894903 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:48.894814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" event={"ID":"44a10a91-5fe6-4150-9567-874fdf632116","Type":"ContainerStarted","Data":"178599e33f3fd11cf407a8b995406f9ee591a80a9f26e01cb03e19c9636b1978"} Jul 14 19:52:48.895317 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:48.894934 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:48.916962 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:48.916905 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" podStartSLOduration=1.453592685 podStartE2EDuration="2.916887488s" totalImagesPullingTime="1.463294803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:48.913729008 +0000 UTC m=+318.758065883" watchObservedRunningTime="2026-07-14 19:52:48.916887488 +0000 UTC m=+318.761224364" Jul 14 19:52:54.876214 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:54.876174 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-5b6jd" Jul 14 19:52:56.474977 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.474944 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb"] Jul 14 19:52:56.484079 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.484056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.487236 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.487209 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:52:56.487361 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.487297 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 19:52:56.488020 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.487998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b4ncl\"" Jul 14 19:52:56.488405 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.488378 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:52:56.489414 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.489390 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb"] Jul 14 19:52:56.509541 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250b0041-6ff0-4dff-b146-b2e02e122934-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509648 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509648 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmr48\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-kube-api-access-xmr48\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509648 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250b0041-6ff0-4dff-b146-b2e02e122934-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.509866 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.509774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610189 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610189 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250b0041-6ff0-4dff-b146-b2e02e122934-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250b0041-6ff0-4dff-b146-b2e02e122934-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmr48\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-kube-api-access-xmr48\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610438 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610773 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610773 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610773 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.610773 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.610745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.611125 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.611100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250b0041-6ff0-4dff-b146-b2e02e122934-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.611235 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.611213 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.612917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.612888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250b0041-6ff0-4dff-b146-b2e02e122934-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.613106 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.613056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250b0041-6ff0-4dff-b146-b2e02e122934-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.620508 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.620445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.620806 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.620783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmr48\" (UniqueName: \"kubernetes.io/projected/250b0041-6ff0-4dff-b146-b2e02e122934-kube-api-access-xmr48\") pod \"data-science-gateway-data-science-gateway-class-7f964fb67884mdb\" (UID: \"250b0041-6ff0-4dff-b146-b2e02e122934\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.795258 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.795170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:52:56.935479 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:56.935450 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb"] Jul 14 19:52:56.936873 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:52:56.936842 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250b0041_6ff0_4dff_b146_b2e02e122934.slice/crio-67a3f4623db798e926b5a6a03c2fc4ac4cd13d2eac25f719c5630ecab81caf33 WatchSource:0}: Error finding container 67a3f4623db798e926b5a6a03c2fc4ac4cd13d2eac25f719c5630ecab81caf33: Status 404 returned error can't find the container with id 67a3f4623db798e926b5a6a03c2fc4ac4cd13d2eac25f719c5630ecab81caf33 Jul 14 19:52:57.940676 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:57.940635 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" event={"ID":"250b0041-6ff0-4dff-b146-b2e02e122934","Type":"ContainerStarted","Data":"67a3f4623db798e926b5a6a03c2fc4ac4cd13d2eac25f719c5630ecab81caf33"} Jul 14 19:52:59.340377 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.340338 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:52:59.340656 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.340416 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:52:59.340656 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.340447 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:52:59.902297 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.902269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77d6d59f96-62vfx" Jul 14 19:52:59.951791 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.951761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" event={"ID":"250b0041-6ff0-4dff-b146-b2e02e122934","Type":"ContainerStarted","Data":"4de30f8032b43a71f86bd3a5c8b9b8e3a1489a56566c0254a002a4c57fd1cf66"} Jul 14 19:52:59.973263 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:52:59.973207 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" podStartSLOduration=1.571962624 podStartE2EDuration="3.97319283s" totalImagesPullingTime="2.401230206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:59.971478027 +0000 UTC m=+329.815814902" watchObservedRunningTime="2026-07-14 19:52:59.97319283 +0000 UTC m=+329.817529704" Jul 14 19:53:00.796054 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:00.796012 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:53:00.801190 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:00.801164 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:53:00.956763 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:00.956733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:53:00.957868 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:00.957848 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb67884mdb" Jul 14 19:53:40.153070 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.153035 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:40.163418 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.163393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:40.164915 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.164886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:40.166423 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.166375 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:53:40.166651 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.166517 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:53:40.167740 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.167722 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fkn4x\"" Jul 14 19:53:40.261348 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.261319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvp8\" (UniqueName: \"kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8\") pod \"kuadrant-operator-catalog-fkzv8\" (UID: \"40421459-fb50-4ae3-aa48-0d40c7702aa2\") " pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:40.362588 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.362548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvp8\" (UniqueName: \"kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8\") pod \"kuadrant-operator-catalog-fkzv8\" (UID: \"40421459-fb50-4ae3-aa48-0d40c7702aa2\") " pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:40.371984 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.371965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvp8\" (UniqueName: \"kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8\") pod \"kuadrant-operator-catalog-fkzv8\" (UID: \"40421459-fb50-4ae3-aa48-0d40c7702aa2\") " pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:40.475224 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.475202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:40.514574 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.514542 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:40.611362 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.611331 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:40.614365 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:40.614333 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40421459_fb50_4ae3_aa48_0d40c7702aa2.slice/crio-9f2e0a209ab34e8f5430ac2b4f97c619e88bf514c8a0203bf743d753630dd5f3 WatchSource:0}: Error finding container 9f2e0a209ab34e8f5430ac2b4f97c619e88bf514c8a0203bf743d753630dd5f3: Status 404 returned error can't find the container with id 9f2e0a209ab34e8f5430ac2b4f97c619e88bf514c8a0203bf743d753630dd5f3 Jul 14 19:53:40.720278 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.720248 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr9b6"] Jul 14 19:53:40.725000 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.724984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:40.731214 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.731166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr9b6"] Jul 14 19:53:40.867379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.867346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpsl6\" (UniqueName: \"kubernetes.io/projected/a45b28b9-2855-4a5b-a776-fea8c7069c98-kube-api-access-hpsl6\") pod \"kuadrant-operator-catalog-dr9b6\" (UID: \"a45b28b9-2855-4a5b-a776-fea8c7069c98\") " pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:40.967890 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.967853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpsl6\" (UniqueName: \"kubernetes.io/projected/a45b28b9-2855-4a5b-a776-fea8c7069c98-kube-api-access-hpsl6\") pod \"kuadrant-operator-catalog-dr9b6\" (UID: \"a45b28b9-2855-4a5b-a776-fea8c7069c98\") " pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:40.976870 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:40.976840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpsl6\" (UniqueName: \"kubernetes.io/projected/a45b28b9-2855-4a5b-a776-fea8c7069c98-kube-api-access-hpsl6\") pod \"kuadrant-operator-catalog-dr9b6\" (UID: \"a45b28b9-2855-4a5b-a776-fea8c7069c98\") " pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:41.035544 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:41.035463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:41.134461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:41.134422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" event={"ID":"40421459-fb50-4ae3-aa48-0d40c7702aa2","Type":"ContainerStarted","Data":"9f2e0a209ab34e8f5430ac2b4f97c619e88bf514c8a0203bf743d753630dd5f3"} Jul 14 19:53:41.172387 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:41.172349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dr9b6"] Jul 14 19:53:41.174126 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:41.174103 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45b28b9_2855_4a5b_a776_fea8c7069c98.slice/crio-f0385f96570bcaa8451267d1eb7b9d874ee3853c400489f0e635e9071b4e2f72 WatchSource:0}: Error finding container f0385f96570bcaa8451267d1eb7b9d874ee3853c400489f0e635e9071b4e2f72: Status 404 returned error can't find the container with id f0385f96570bcaa8451267d1eb7b9d874ee3853c400489f0e635e9071b4e2f72 Jul 14 19:53:42.141901 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:42.141862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" event={"ID":"a45b28b9-2855-4a5b-a776-fea8c7069c98","Type":"ContainerStarted","Data":"f0385f96570bcaa8451267d1eb7b9d874ee3853c400489f0e635e9071b4e2f72"} Jul 14 19:53:43.147918 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.147878 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" event={"ID":"40421459-fb50-4ae3-aa48-0d40c7702aa2","Type":"ContainerStarted","Data":"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833"} Jul 14 19:53:43.148343 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.147988 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" podUID="40421459-fb50-4ae3-aa48-0d40c7702aa2" containerName="registry-server" containerID="cri-o://2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833" gracePeriod=2 Jul 14 19:53:43.149510 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.149471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" event={"ID":"a45b28b9-2855-4a5b-a776-fea8c7069c98","Type":"ContainerStarted","Data":"74898ed599ccc4a7cfbb40fc96b478300f10aab36fa70edb3a5c630d6d22e1ba"} Jul 14 19:53:43.166022 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.165980 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" podStartSLOduration=1.081281296 podStartE2EDuration="3.165965337s" totalImagesPullingTime="2.084684041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:43.163669447 +0000 UTC m=+373.008006317" watchObservedRunningTime="2026-07-14 19:53:43.165965337 +0000 UTC m=+373.010302211" Jul 14 19:53:43.178402 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.178358 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" podStartSLOduration=1.652727558 podStartE2EDuration="3.17834309s" totalImagesPullingTime="1.525615532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:43.177733766 +0000 UTC m=+373.022070643" watchObservedRunningTime="2026-07-14 19:53:43.17834309 +0000 UTC m=+373.022679963" Jul 14 19:53:43.404083 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.404030 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:43.490637 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.490609 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xvp8\" (UniqueName: \"kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8\") pod \"40421459-fb50-4ae3-aa48-0d40c7702aa2\" (UID: \"40421459-fb50-4ae3-aa48-0d40c7702aa2\") " Jul 14 19:53:43.492828 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.492807 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8" (OuterVolumeSpecName: "kube-api-access-7xvp8") pod "40421459-fb50-4ae3-aa48-0d40c7702aa2" (UID: "40421459-fb50-4ae3-aa48-0d40c7702aa2"). InnerVolumeSpecName "kube-api-access-7xvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:43.591601 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:43.591577 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xvp8\" (UniqueName: \"kubernetes.io/projected/40421459-fb50-4ae3-aa48-0d40c7702aa2-kube-api-access-7xvp8\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:53:44.155620 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.155534 2574 generic.go:356] "Generic (PLEG): container finished" podID="40421459-fb50-4ae3-aa48-0d40c7702aa2" containerID="2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833" exitCode=0 Jul 14 19:53:44.155620 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.155578 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" Jul 14 19:53:44.156014 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.155617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" event={"ID":"40421459-fb50-4ae3-aa48-0d40c7702aa2","Type":"ContainerDied","Data":"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833"} Jul 14 19:53:44.156014 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.155664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fkzv8" event={"ID":"40421459-fb50-4ae3-aa48-0d40c7702aa2","Type":"ContainerDied","Data":"9f2e0a209ab34e8f5430ac2b4f97c619e88bf514c8a0203bf743d753630dd5f3"} Jul 14 19:53:44.156014 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.155691 2574 scope.go:117] "RemoveContainer" containerID="2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833" Jul 14 19:53:44.165336 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.165314 2574 scope.go:117] "RemoveContainer" containerID="2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833" Jul 14 19:53:44.165633 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:53:44.165612 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833\": container with ID starting with 2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833 not found: ID does not exist" containerID="2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833" Jul 14 19:53:44.165694 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.165641 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833"} err="failed to get container status \"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833\": rpc error: code = NotFound desc = could not find container \"2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833\": container with ID starting with 2954e3d11b87f0d485f998931f3789ef689e70923afb22580d68d4d281bbc833 not found: ID does not exist" Jul 14 19:53:44.177034 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.177013 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:44.181095 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.181075 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fkzv8"] Jul 14 19:53:44.598849 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:44.598814 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40421459-fb50-4ae3-aa48-0d40c7702aa2" path="/var/lib/kubelet/pods/40421459-fb50-4ae3-aa48-0d40c7702aa2/volumes" Jul 14 19:53:51.035592 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:51.035545 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:51.036085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:51.035614 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:51.058814 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:51.058784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:51.208025 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:51.208000 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dr9b6" Jul 14 19:53:55.958091 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.958056 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82"] Jul 14 19:53:55.958465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.958388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40421459-fb50-4ae3-aa48-0d40c7702aa2" containerName="registry-server" Jul 14 19:53:55.958465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.958398 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40421459-fb50-4ae3-aa48-0d40c7702aa2" containerName="registry-server" Jul 14 19:53:55.958465 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.958446 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40421459-fb50-4ae3-aa48-0d40c7702aa2" containerName="registry-server" Jul 14 19:53:55.961474 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.961457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:55.964088 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.964070 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7c6z\"" Jul 14 19:53:55.970176 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:55.970148 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82"] Jul 14 19:53:56.095451 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.095419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.095657 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.095484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.095657 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.095542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hnb\" (UniqueName: \"kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.196625 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.196591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.196625 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.196629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.196839 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.196663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hnb\" (UniqueName: \"kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.196941 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.196920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.197030 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.197011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.205551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.205527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hnb\" (UniqueName: \"kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.271379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.271320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:53:56.422729 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.422705 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82"] Jul 14 19:53:56.425280 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:56.425244 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb8426a_e05d_4ddf_9fda_ed7df477b963.slice/crio-6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525 WatchSource:0}: Error finding container 6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525: Status 404 returned error can't find the container with id 6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525 Jul 14 19:53:56.756173 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.756141 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z"] Jul 14 19:53:56.759473 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.759457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:56.768639 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.768610 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z"] Jul 14 19:53:56.902013 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.901935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:56.902013 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.901983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:56.902179 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:56.902033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987lv\" (UniqueName: \"kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.003331 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.003303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.003671 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.003352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.003671 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.003466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987lv\" (UniqueName: \"kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.003746 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.003735 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.003800 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.003780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.012203 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.012184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-987lv\" (UniqueName: \"kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.069043 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.069018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:53:57.197089 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.197057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z"] Jul 14 19:53:57.200766 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:57.200735 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c96c264_6037_45c8_8727_7c9991b9fa17.slice/crio-052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343 WatchSource:0}: Error finding container 052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343: Status 404 returned error can't find the container with id 052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343 Jul 14 19:53:57.214704 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.214675 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerID="705b007eaefe44850066af1d094c5c6a7180ef3689d4274ce75a785c02dfe7f2" exitCode=0 Jul 14 19:53:57.214929 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.214771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" event={"ID":"cbb8426a-e05d-4ddf-9fda-ed7df477b963","Type":"ContainerDied","Data":"705b007eaefe44850066af1d094c5c6a7180ef3689d4274ce75a785c02dfe7f2"} Jul 14 19:53:57.214929 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.214829 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" event={"ID":"cbb8426a-e05d-4ddf-9fda-ed7df477b963","Type":"ContainerStarted","Data":"6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525"} Jul 14 19:53:57.216478 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.216442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" event={"ID":"7c96c264-6037-45c8-8727-7c9991b9fa17","Type":"ContainerStarted","Data":"052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343"} Jul 14 19:53:57.437066 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.437034 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws"] Jul 14 19:53:57.440379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.440364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.449776 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.449726 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws"] Jul 14 19:53:57.608611 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.608578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.608789 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.608634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.608789 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.608733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skbf\" (UniqueName: \"kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.709205 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.709177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.709325 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.709239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skbf\" (UniqueName: \"kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.709325 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.709297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.709628 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.709607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.709695 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.709611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.718548 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.718523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skbf\" (UniqueName: \"kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.750221 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.750200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:53:57.760594 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.760565 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd"] Jul 14 19:53:57.764654 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.764636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:57.773005 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.772952 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd"] Jul 14 19:53:57.887436 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.887409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws"] Jul 14 19:53:57.889104 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:57.889074 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf1d59c_b932_4641_92e3_3ed3258fcfcf.slice/crio-fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0 WatchSource:0}: Error finding container fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0: Status 404 returned error can't find the container with id fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0 Jul 14 19:53:57.911557 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.911534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:57.911639 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.911570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:57.911694 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:57.911677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k499\" (UniqueName: \"kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.012658 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.012627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.012977 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.012666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.012977 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.012742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k499\" (UniqueName: \"kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.013056 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.012993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.013056 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.013013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.021284 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.021257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k499\" (UniqueName: \"kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.077301 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.077269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:53:58.211195 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.211166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd"] Jul 14 19:53:58.212688 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:53:58.212661 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fb17c4_864d_4661_b455_b2ed61c238f1.slice/crio-4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78 WatchSource:0}: Error finding container 4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78: Status 404 returned error can't find the container with id 4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78 Jul 14 19:53:58.224226 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.224194 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerID="d9efd9cd7294426e1cef4e6fde71666e2cce31676e5d784c49d2458dcd4b5eb6" exitCode=0 Jul 14 19:53:58.224337 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.224280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" event={"ID":"cbb8426a-e05d-4ddf-9fda-ed7df477b963","Type":"ContainerDied","Data":"d9efd9cd7294426e1cef4e6fde71666e2cce31676e5d784c49d2458dcd4b5eb6"} Jul 14 19:53:58.225720 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.225697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" event={"ID":"27fb17c4-864d-4661-b455-b2ed61c238f1","Type":"ContainerStarted","Data":"4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78"} Jul 14 19:53:58.227305 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.227247 2574 generic.go:356] "Generic (PLEG): container finished" podID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerID="1f68581abdb94eefa910ddc41e2696ad45fc0d467f0b9709b752c4b5f447f32d" exitCode=0 Jul 14 19:53:58.227385 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.227333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" event={"ID":"7c96c264-6037-45c8-8727-7c9991b9fa17","Type":"ContainerDied","Data":"1f68581abdb94eefa910ddc41e2696ad45fc0d467f0b9709b752c4b5f447f32d"} Jul 14 19:53:58.228918 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.228897 2574 generic.go:356] "Generic (PLEG): container finished" podID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerID="346c981827ebaf6c3deb189983de9a04976057eb4ce2ec4aec76f29158e12f24" exitCode=0 Jul 14 19:53:58.228973 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.228943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" event={"ID":"baf1d59c-b932-4641-92e3-3ed3258fcfcf","Type":"ContainerDied","Data":"346c981827ebaf6c3deb189983de9a04976057eb4ce2ec4aec76f29158e12f24"} Jul 14 19:53:58.229011 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:58.228977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" event={"ID":"baf1d59c-b932-4641-92e3-3ed3258fcfcf","Type":"ContainerStarted","Data":"fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0"} Jul 14 19:53:59.237043 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.237013 2574 generic.go:356] "Generic (PLEG): container finished" podID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerID="9e9934b5e9a73eec7d88209eb08035fed66d91ac1f97e3e6399026fe2a92868a" exitCode=0 Jul 14 19:53:59.237423 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.237115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" event={"ID":"baf1d59c-b932-4641-92e3-3ed3258fcfcf","Type":"ContainerDied","Data":"9e9934b5e9a73eec7d88209eb08035fed66d91ac1f97e3e6399026fe2a92868a"} Jul 14 19:53:59.239425 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.239403 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerID="a6754d0087781c84de8770b57ba20d11cf30e9bbdbc03857989134f9e76e437d" exitCode=0 Jul 14 19:53:59.239521 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.239448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" event={"ID":"cbb8426a-e05d-4ddf-9fda-ed7df477b963","Type":"ContainerDied","Data":"a6754d0087781c84de8770b57ba20d11cf30e9bbdbc03857989134f9e76e437d"} Jul 14 19:53:59.240845 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.240823 2574 generic.go:356] "Generic (PLEG): container finished" podID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerID="7d14531b50d2e5bc3c05777fe97f46972a63db491f4fde8c2cb42f75ee91ef9d" exitCode=0 Jul 14 19:53:59.240925 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:53:59.240863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" event={"ID":"27fb17c4-864d-4661-b455-b2ed61c238f1","Type":"ContainerDied","Data":"7d14531b50d2e5bc3c05777fe97f46972a63db491f4fde8c2cb42f75ee91ef9d"} Jul 14 19:54:00.248821 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.248780 2574 generic.go:356] "Generic (PLEG): container finished" podID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerID="080b24d25e447535592196fc3b76d641f26baf34f52b00f560ed654ccd6e20dc" exitCode=0 Jul 14 19:54:00.249187 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.248860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" event={"ID":"baf1d59c-b932-4641-92e3-3ed3258fcfcf","Type":"ContainerDied","Data":"080b24d25e447535592196fc3b76d641f26baf34f52b00f560ed654ccd6e20dc"} Jul 14 19:54:00.250855 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.250834 2574 generic.go:356] "Generic (PLEG): container finished" podID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerID="2f56607bb77e662458c94a72c3719b9b5d91372d4f50fba1bc20519a7da0dc9d" exitCode=0 Jul 14 19:54:00.250983 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.250918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" event={"ID":"27fb17c4-864d-4661-b455-b2ed61c238f1","Type":"ContainerDied","Data":"2f56607bb77e662458c94a72c3719b9b5d91372d4f50fba1bc20519a7da0dc9d"} Jul 14 19:54:00.252777 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.252758 2574 generic.go:356] "Generic (PLEG): container finished" podID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerID="988edc11f441ae261d89bd763f86b17f527b18704fe9e7f04122b07808a8a0f5" exitCode=0 Jul 14 19:54:00.252874 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.252825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" event={"ID":"7c96c264-6037-45c8-8727-7c9991b9fa17","Type":"ContainerDied","Data":"988edc11f441ae261d89bd763f86b17f527b18704fe9e7f04122b07808a8a0f5"} Jul 14 19:54:00.395370 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.395293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:54:00.433746 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.433705 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle\") pod \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " Jul 14 19:54:00.433896 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.433763 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2hnb\" (UniqueName: \"kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb\") pod \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " Jul 14 19:54:00.433896 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.433824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util\") pod \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\" (UID: \"cbb8426a-e05d-4ddf-9fda-ed7df477b963\") " Jul 14 19:54:00.434198 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.434170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle" (OuterVolumeSpecName: "bundle") pod "cbb8426a-e05d-4ddf-9fda-ed7df477b963" (UID: "cbb8426a-e05d-4ddf-9fda-ed7df477b963"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:00.435904 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.435877 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb" (OuterVolumeSpecName: "kube-api-access-h2hnb") pod "cbb8426a-e05d-4ddf-9fda-ed7df477b963" (UID: "cbb8426a-e05d-4ddf-9fda-ed7df477b963"). InnerVolumeSpecName "kube-api-access-h2hnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:00.439276 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.439255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util" (OuterVolumeSpecName: "util") pod "cbb8426a-e05d-4ddf-9fda-ed7df477b963" (UID: "cbb8426a-e05d-4ddf-9fda-ed7df477b963"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:00.535236 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.535164 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2hnb\" (UniqueName: \"kubernetes.io/projected/cbb8426a-e05d-4ddf-9fda-ed7df477b963-kube-api-access-h2hnb\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:00.535236 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.535188 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:00.535236 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:00.535198 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbb8426a-e05d-4ddf-9fda-ed7df477b963-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:01.260150 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.260115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" event={"ID":"cbb8426a-e05d-4ddf-9fda-ed7df477b963","Type":"ContainerDied","Data":"6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525"} Jul 14 19:54:01.260645 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.260177 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e7e28506ad88e27a01cdcf704ea5b0839e854c0d9550bee8e14b5a85a09f525" Jul 14 19:54:01.260645 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.260133 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82" Jul 14 19:54:01.262531 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.262483 2574 generic.go:356] "Generic (PLEG): container finished" podID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerID="1826ed22bed102d46bf8f0ef64ae99203fd0c47b663f0cfcbb16cc3c573ba539" exitCode=0 Jul 14 19:54:01.262665 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.262590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" event={"ID":"27fb17c4-864d-4661-b455-b2ed61c238f1","Type":"ContainerDied","Data":"1826ed22bed102d46bf8f0ef64ae99203fd0c47b663f0cfcbb16cc3c573ba539"} Jul 14 19:54:01.264649 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.264627 2574 generic.go:356] "Generic (PLEG): container finished" podID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerID="b146b2c609848c65ee6f0ed8a062362eb288642b55ed62759591cc39fe2f210a" exitCode=0 Jul 14 19:54:01.264747 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.264694 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" event={"ID":"7c96c264-6037-45c8-8727-7c9991b9fa17","Type":"ContainerDied","Data":"b146b2c609848c65ee6f0ed8a062362eb288642b55ed62759591cc39fe2f210a"} Jul 14 19:54:01.394481 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.394459 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:54:01.440948 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.440923 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util\") pod \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " Jul 14 19:54:01.441136 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.440957 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle\") pod \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " Jul 14 19:54:01.441136 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.440995 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4skbf\" (UniqueName: \"kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf\") pod \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\" (UID: \"baf1d59c-b932-4641-92e3-3ed3258fcfcf\") " Jul 14 19:54:01.441587 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.441546 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle" (OuterVolumeSpecName: "bundle") pod "baf1d59c-b932-4641-92e3-3ed3258fcfcf" (UID: "baf1d59c-b932-4641-92e3-3ed3258fcfcf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:01.443328 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.443293 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf" (OuterVolumeSpecName: "kube-api-access-4skbf") pod "baf1d59c-b932-4641-92e3-3ed3258fcfcf" (UID: "baf1d59c-b932-4641-92e3-3ed3258fcfcf"). InnerVolumeSpecName "kube-api-access-4skbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:01.447711 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.447676 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util" (OuterVolumeSpecName: "util") pod "baf1d59c-b932-4641-92e3-3ed3258fcfcf" (UID: "baf1d59c-b932-4641-92e3-3ed3258fcfcf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:01.542551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.542427 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:01.542551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.542469 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/baf1d59c-b932-4641-92e3-3ed3258fcfcf-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:01.542551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:01.542478 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skbf\" (UniqueName: \"kubernetes.io/projected/baf1d59c-b932-4641-92e3-3ed3258fcfcf-kube-api-access-4skbf\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.271205 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.271169 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" Jul 14 19:54:02.271205 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.271191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws" event={"ID":"baf1d59c-b932-4641-92e3-3ed3258fcfcf","Type":"ContainerDied","Data":"fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0"} Jul 14 19:54:02.271640 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.271241 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc9a3d97f53b3365dbf1339d2baea4447bf9ecea89c9d5f3a5dcc70bfc8d64b0" Jul 14 19:54:02.403634 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.403609 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:54:02.435147 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.435125 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:54:02.452226 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452203 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle\") pod \"27fb17c4-864d-4661-b455-b2ed61c238f1\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " Jul 14 19:54:02.452350 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452235 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util\") pod \"7c96c264-6037-45c8-8727-7c9991b9fa17\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " Jul 14 19:54:02.452350 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k499\" (UniqueName: \"kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499\") pod \"27fb17c4-864d-4661-b455-b2ed61c238f1\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " Jul 14 19:54:02.452350 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452279 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util\") pod \"27fb17c4-864d-4661-b455-b2ed61c238f1\" (UID: \"27fb17c4-864d-4661-b455-b2ed61c238f1\") " Jul 14 19:54:02.452350 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452312 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle\") pod \"7c96c264-6037-45c8-8727-7c9991b9fa17\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " Jul 14 19:54:02.453155 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.452353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-987lv\" (UniqueName: \"kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv\") pod \"7c96c264-6037-45c8-8727-7c9991b9fa17\" (UID: \"7c96c264-6037-45c8-8727-7c9991b9fa17\") " Jul 14 19:54:02.453398 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.453240 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle" (OuterVolumeSpecName: "bundle") pod "27fb17c4-864d-4661-b455-b2ed61c238f1" (UID: "27fb17c4-864d-4661-b455-b2ed61c238f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:02.453614 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.453579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle" (OuterVolumeSpecName: "bundle") pod "7c96c264-6037-45c8-8727-7c9991b9fa17" (UID: "7c96c264-6037-45c8-8727-7c9991b9fa17"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:02.455441 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.455400 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499" (OuterVolumeSpecName: "kube-api-access-2k499") pod "27fb17c4-864d-4661-b455-b2ed61c238f1" (UID: "27fb17c4-864d-4661-b455-b2ed61c238f1"). InnerVolumeSpecName "kube-api-access-2k499". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:02.457254 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.457235 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.457383 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.457369 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k499\" (UniqueName: \"kubernetes.io/projected/27fb17c4-864d-4661-b455-b2ed61c238f1-kube-api-access-2k499\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.457695 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.457659 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-bundle\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.457695 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.457616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv" (OuterVolumeSpecName: "kube-api-access-987lv") pod "7c96c264-6037-45c8-8727-7c9991b9fa17" (UID: "7c96c264-6037-45c8-8727-7c9991b9fa17"). InnerVolumeSpecName "kube-api-access-987lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:02.460018 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.459992 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util" (OuterVolumeSpecName: "util") pod "27fb17c4-864d-4661-b455-b2ed61c238f1" (UID: "27fb17c4-864d-4661-b455-b2ed61c238f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:02.461326 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.461301 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util" (OuterVolumeSpecName: "util") pod "7c96c264-6037-45c8-8727-7c9991b9fa17" (UID: "7c96c264-6037-45c8-8727-7c9991b9fa17"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:02.558551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.558468 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c96c264-6037-45c8-8727-7c9991b9fa17-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.558551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.558521 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27fb17c4-864d-4661-b455-b2ed61c238f1-util\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:02.558551 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:02.558531 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-987lv\" (UniqueName: \"kubernetes.io/projected/7c96c264-6037-45c8-8727-7c9991b9fa17-kube-api-access-987lv\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:03.277799 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.277770 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" Jul 14 19:54:03.277799 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.277781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd" event={"ID":"27fb17c4-864d-4661-b455-b2ed61c238f1","Type":"ContainerDied","Data":"4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78"} Jul 14 19:54:03.278283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.277872 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e152501c13bf1bea59c4181187d138c1906354fea4a42b92863d99a23f26b78" Jul 14 19:54:03.279904 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.279881 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" Jul 14 19:54:03.280044 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.279944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z" event={"ID":"7c96c264-6037-45c8-8727-7c9991b9fa17","Type":"ContainerDied","Data":"052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343"} Jul 14 19:54:03.280044 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:03.280008 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052139673e9e045e2f4db4ad3a4ebd2b7848e134efcdc91424901202f6f7c343" Jul 14 19:54:13.145522 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145471 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q"] Jul 14 19:54:13.145941 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145887 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="util" Jul 14 19:54:13.145941 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145906 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="util" Jul 14 19:54:13.145941 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145927 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="pull" Jul 14 19:54:13.145941 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145937 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="pull" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145954 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145964 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145975 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.145991 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146001 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="pull" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146007 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="pull" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146022 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146031 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146055 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146065 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146079 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="util" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146091 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146096 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146102 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146106 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="extract" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146117 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="pull" Jul 14 19:54:13.146114 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146122 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="pull" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146132 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="pull" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146137 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="pull" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146222 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf1d59c-b932-4641-92e3-3ed3258fcfcf" containerName="extract" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146234 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb8426a-e05d-4ddf-9fda-ed7df477b963" containerName="extract" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146246 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fb17c4-864d-4661-b455-b2ed61c238f1" containerName="extract" Jul 14 19:54:13.146670 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.146266 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c96c264-6037-45c8-8727-7c9991b9fa17" containerName="extract" Jul 14 19:54:13.150813 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.150792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:13.153342 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.153322 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kb2k8\"" Jul 14 19:54:13.157307 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.157279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q"] Jul 14 19:54:13.249857 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.249826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jckch\" (UniqueName: \"kubernetes.io/projected/647410ee-9e16-4eed-a54c-27421157a0d4-kube-api-access-jckch\") pod \"authorino-operator-75cc8d5cb7-zkq2q\" (UID: \"647410ee-9e16-4eed-a54c-27421157a0d4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:13.350417 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.350385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jckch\" (UniqueName: \"kubernetes.io/projected/647410ee-9e16-4eed-a54c-27421157a0d4-kube-api-access-jckch\") pod \"authorino-operator-75cc8d5cb7-zkq2q\" (UID: \"647410ee-9e16-4eed-a54c-27421157a0d4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:13.362766 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.362743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jckch\" (UniqueName: \"kubernetes.io/projected/647410ee-9e16-4eed-a54c-27421157a0d4-kube-api-access-jckch\") pod \"authorino-operator-75cc8d5cb7-zkq2q\" (UID: \"647410ee-9e16-4eed-a54c-27421157a0d4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:13.466714 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.466691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:13.605762 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:13.605740 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q"] Jul 14 19:54:13.608040 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:54:13.608017 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647410ee_9e16_4eed_a54c_27421157a0d4.slice/crio-d95b72e7c376f94ff5c7d50855d74ec1ea60e44f8f4df71280acb1ba2a3f34b7 WatchSource:0}: Error finding container d95b72e7c376f94ff5c7d50855d74ec1ea60e44f8f4df71280acb1ba2a3f34b7: Status 404 returned error can't find the container with id d95b72e7c376f94ff5c7d50855d74ec1ea60e44f8f4df71280acb1ba2a3f34b7 Jul 14 19:54:14.334124 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:14.334088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" event={"ID":"647410ee-9e16-4eed-a54c-27421157a0d4","Type":"ContainerStarted","Data":"d95b72e7c376f94ff5c7d50855d74ec1ea60e44f8f4df71280acb1ba2a3f34b7"} Jul 14 19:54:16.346012 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:16.345976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" event={"ID":"647410ee-9e16-4eed-a54c-27421157a0d4","Type":"ContainerStarted","Data":"58d8bfa84106cbf329472bb4b2b2b50d82ec71c5357c4276d705b3176416a113"} Jul 14 19:54:16.346349 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:16.346124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:16.366779 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:16.366726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" podStartSLOduration=0.778459878 podStartE2EDuration="3.366713777s" totalImagesPullingTime="2.588253899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:16.364394202 +0000 UTC m=+406.208731078" watchObservedRunningTime="2026-07-14 19:54:16.366713777 +0000 UTC m=+406.211050651" Jul 14 19:54:18.324317 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.324283 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w"] Jul 14 19:54:18.330725 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.330705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.349292 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.349269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w"] Jul 14 19:54:18.350757 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.350734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-65fsx\"" Jul 14 19:54:18.393833 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.393804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6697\" (UniqueName: \"kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.393945 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.393841 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.495270 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.495233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6697\" (UniqueName: \"kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.495409 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.495281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.495640 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.495625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.504348 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.504329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6697\" (UniqueName: \"kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697\") pod \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.641071 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.640998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:18.779274 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:18.779249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w"] Jul 14 19:54:18.781379 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:54:18.781343 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1162b1d8_cc95_4762_bfd3_5de7bc624a6b.slice/crio-4f970214a74f30f919581939a4f68b46283b0d3e3efc92044fcc2acdb1a6ac77 WatchSource:0}: Error finding container 4f970214a74f30f919581939a4f68b46283b0d3e3efc92044fcc2acdb1a6ac77: Status 404 returned error can't find the container with id 4f970214a74f30f919581939a4f68b46283b0d3e3efc92044fcc2acdb1a6ac77 Jul 14 19:54:19.361085 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:19.361043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" event={"ID":"1162b1d8-cc95-4762-bfd3-5de7bc624a6b","Type":"ContainerStarted","Data":"4f970214a74f30f919581939a4f68b46283b0d3e3efc92044fcc2acdb1a6ac77"} Jul 14 19:54:24.397457 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:24.397364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" event={"ID":"1162b1d8-cc95-4762-bfd3-5de7bc624a6b","Type":"ContainerStarted","Data":"0f6b3f34389e3218218688ab46309359a1e57d1866909e69e8b0688be7812dec"} Jul 14 19:54:24.397837 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:24.397524 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:24.418441 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:24.418385 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" podStartSLOduration=1.22187261 podStartE2EDuration="6.418370458s" totalImagesPullingTime="5.196497848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:24.417556792 +0000 UTC m=+414.261893660" watchObservedRunningTime="2026-07-14 19:54:24.418370458 +0000 UTC m=+414.262707332" Jul 14 19:54:27.352833 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:27.352794 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zkq2q" Jul 14 19:54:35.405582 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:35.405551 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:36.608047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.608013 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72"] Jul 14 19:54:36.612875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.612855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.623745 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.623722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72"] Jul 14 19:54:36.724728 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.724700 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72"] Jul 14 19:54:36.724949 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:36.724929 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2bpf8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" Jul 14 19:54:36.755661 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.755634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.755793 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.755691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpf8\" (UniqueName: \"kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.856826 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.856799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.856968 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.856838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpf8\" (UniqueName: \"kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.857156 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.857136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:36.866724 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:36.866661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpf8\" (UniqueName: \"kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.261808 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.261777 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72"] Jul 14 19:54:37.276852 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.276824 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72"] Jul 14 19:54:37.287174 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.287146 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w"] Jul 14 19:54:37.287375 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.287349 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" containerName="manager" containerID="cri-o://0f6b3f34389e3218218688ab46309359a1e57d1866909e69e8b0688be7812dec" gracePeriod=2 Jul 14 19:54:37.294220 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.294193 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w"] Jul 14 19:54:37.313762 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.313737 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:54:37.314308 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.314287 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" containerName="manager" Jul 14 19:54:37.314308 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.314310 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" containerName="manager" Jul 14 19:54:37.315226 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.314411 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" containerName="manager" Jul 14 19:54:37.317794 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.317775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.329618 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.329595 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:54:37.352259 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.352227 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:37.462097 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.462068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2pf\" (UniqueName: \"kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.462247 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.462113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.464922 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.464891 2574 generic.go:356] "Generic (PLEG): container finished" podID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" containerID="0f6b3f34389e3218218688ab46309359a1e57d1866909e69e8b0688be7812dec" exitCode=0 Jul 14 19:54:37.465083 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.465068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.467428 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.467396 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.469373 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.469339 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:37.472373 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.472347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.474667 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.474637 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.482270 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.482233 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:37.537538 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.537518 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:37.539889 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.539866 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:37.541726 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:37.541706 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:37.563021 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.562997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume\") pod \"e4b5dc37-c39e-4717-ade7-9c804b30673d\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " Jul 14 19:54:37.563121 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.563030 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bpf8\" (UniqueName: \"kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8\") pod \"e4b5dc37-c39e-4717-ade7-9c804b30673d\" (UID: \"e4b5dc37-c39e-4717-ade7-9c804b30673d\") " Jul 14 19:54:37.563210 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.563196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2pf\" (UniqueName: \"kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.563274 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.563225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.563325 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.563273 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e4b5dc37-c39e-4717-ade7-9c804b30673d" (UID: "e4b5dc37-c39e-4717-ade7-9c804b30673d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:37.563563 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.563540 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.565206 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.565188 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8" (OuterVolumeSpecName: "kube-api-access-2bpf8") pod "e4b5dc37-c39e-4717-ade7-9c804b30673d" (UID: "e4b5dc37-c39e-4717-ade7-9c804b30673d"). InnerVolumeSpecName "kube-api-access-2bpf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:37.577283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.577262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2pf\" (UniqueName: \"kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-drt88\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.663712 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.663689 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume\") pod \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " Jul 14 19:54:37.664047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.663781 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6697\" (UniqueName: \"kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697\") pod \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\" (UID: \"1162b1d8-cc95-4762-bfd3-5de7bc624a6b\") " Jul 14 19:54:37.664047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.663982 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4b5dc37-c39e-4717-ade7-9c804b30673d-extensions-socket-volume\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:37.664047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.663998 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bpf8\" (UniqueName: \"kubernetes.io/projected/e4b5dc37-c39e-4717-ade7-9c804b30673d-kube-api-access-2bpf8\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:37.664047 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.664008 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1162b1d8-cc95-4762-bfd3-5de7bc624a6b" (UID: "1162b1d8-cc95-4762-bfd3-5de7bc624a6b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:37.665652 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.665632 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697" (OuterVolumeSpecName: "kube-api-access-c6697") pod "1162b1d8-cc95-4762-bfd3-5de7bc624a6b" (UID: "1162b1d8-cc95-4762-bfd3-5de7bc624a6b"). InnerVolumeSpecName "kube-api-access-c6697". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:37.673164 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.673147 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:37.765379 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.765347 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-extensions-socket-volume\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:37.765553 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.765525 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6697\" (UniqueName: \"kubernetes.io/projected/1162b1d8-cc95-4762-bfd3-5de7bc624a6b-kube-api-access-c6697\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:54:37.814025 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:37.814000 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:54:37.816227 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:54:37.816202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84408311_cbbe_42fa_8996_fa61f62bb538.slice/crio-0771c609605bd22c59df1e2afd0326e2f805f95303b2c0e9ad9488898e222d66 WatchSource:0}: Error finding container 0771c609605bd22c59df1e2afd0326e2f805f95303b2c0e9ad9488898e222d66: Status 404 returned error can't find the container with id 0771c609605bd22c59df1e2afd0326e2f805f95303b2c0e9ad9488898e222d66 Jul 14 19:54:38.471268 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.471233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" event={"ID":"84408311-cbbe-42fa-8996-fa61f62bb538","Type":"ContainerStarted","Data":"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5"} Jul 14 19:54:38.471461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.471293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" event={"ID":"84408311-cbbe-42fa-8996-fa61f62bb538","Type":"ContainerStarted","Data":"0771c609605bd22c59df1e2afd0326e2f805f95303b2c0e9ad9488898e222d66"} Jul 14 19:54:38.471461 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.471328 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:54:38.472637 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.472607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:38.472753 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.472673 2574 scope.go:117] "RemoveContainer" containerID="0f6b3f34389e3218218688ab46309359a1e57d1866909e69e8b0688be7812dec" Jul 14 19:54:38.472795 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.472784 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:38.490690 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.490650 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" podStartSLOduration=1.490636542 podStartE2EDuration="1.490636542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:38.489298235 +0000 UTC m=+428.333635146" watchObservedRunningTime="2026-07-14 19:54:38.490636542 +0000 UTC m=+428.334973479" Jul 14 19:54:38.491247 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:38.491221 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:38.492971 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:38.492947 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:38.494869 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:38.494847 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cmv72\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cmv72" Jul 14 19:54:38.496666 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:54:38.496642 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-5p58w\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-5p58w" Jul 14 19:54:38.599430 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.599397 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1162b1d8-cc95-4762-bfd3-5de7bc624a6b" path="/var/lib/kubelet/pods/1162b1d8-cc95-4762-bfd3-5de7bc624a6b/volumes" Jul 14 19:54:38.599779 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:38.599765 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b5dc37-c39e-4717-ade7-9c804b30673d" path="/var/lib/kubelet/pods/e4b5dc37-c39e-4717-ade7-9c804b30673d/volumes" Jul 14 19:54:49.481042 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:54:49.481009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:55:05.846282 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:05.846204 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:55:05.846694 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:05.846422 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" podUID="84408311-cbbe-42fa-8996-fa61f62bb538" containerName="manager" containerID="cri-o://7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5" gracePeriod=10 Jul 14 19:55:06.089590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.089567 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:55:06.138824 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.138751 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 19:55:06.139066 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.139054 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84408311-cbbe-42fa-8996-fa61f62bb538" containerName="manager" Jul 14 19:55:06.139106 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.139067 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84408311-cbbe-42fa-8996-fa61f62bb538" containerName="manager" Jul 14 19:55:06.139139 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.139123 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84408311-cbbe-42fa-8996-fa61f62bb538" containerName="manager" Jul 14 19:55:06.141556 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.141538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.150542 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.150518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 19:55:06.197235 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume\") pod \"84408311-cbbe-42fa-8996-fa61f62bb538\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " Jul 14 19:55:06.197353 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197310 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p2pf\" (UniqueName: \"kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf\") pod \"84408311-cbbe-42fa-8996-fa61f62bb538\" (UID: \"84408311-cbbe-42fa-8996-fa61f62bb538\") " Jul 14 19:55:06.197450 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmvw\" (UniqueName: \"kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.197575 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.197622 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197601 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "84408311-cbbe-42fa-8996-fa61f62bb538" (UID: "84408311-cbbe-42fa-8996-fa61f62bb538"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:55:06.197709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.197691 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84408311-cbbe-42fa-8996-fa61f62bb538-extensions-socket-volume\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:55:06.199528 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.199482 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf" (OuterVolumeSpecName: "kube-api-access-2p2pf") pod "84408311-cbbe-42fa-8996-fa61f62bb538" (UID: "84408311-cbbe-42fa-8996-fa61f62bb538"). InnerVolumeSpecName "kube-api-access-2p2pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:55:06.298983 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.298952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.299104 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.299018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmvw\" (UniqueName: \"kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.299104 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.299057 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p2pf\" (UniqueName: \"kubernetes.io/projected/84408311-cbbe-42fa-8996-fa61f62bb538-kube-api-access-2p2pf\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:55:06.299304 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.299285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.308053 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.308022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmvw\" (UniqueName: \"kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw\") pod \"kuadrant-operator-controller-manager-86878c769f-nz497\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.452246 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.452214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:06.587408 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.587386 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 19:55:06.589069 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:55:06.589041 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688a0ec9_ea1e_49db_9b49_084856d56e8d.slice/crio-1d86960252da58746bf670ab0d7a99fe09325eedfb574844d84f93df2a499532 WatchSource:0}: Error finding container 1d86960252da58746bf670ab0d7a99fe09325eedfb574844d84f93df2a499532: Status 404 returned error can't find the container with id 1d86960252da58746bf670ab0d7a99fe09325eedfb574844d84f93df2a499532 Jul 14 19:55:06.614474 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.614445 2574 generic.go:356] "Generic (PLEG): container finished" podID="84408311-cbbe-42fa-8996-fa61f62bb538" containerID="7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5" exitCode=0 Jul 14 19:55:06.614590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.614522 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" Jul 14 19:55:06.614590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.614530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" event={"ID":"84408311-cbbe-42fa-8996-fa61f62bb538","Type":"ContainerDied","Data":"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5"} Jul 14 19:55:06.614590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.614583 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88" event={"ID":"84408311-cbbe-42fa-8996-fa61f62bb538","Type":"ContainerDied","Data":"0771c609605bd22c59df1e2afd0326e2f805f95303b2c0e9ad9488898e222d66"} Jul 14 19:55:06.614741 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.614606 2574 scope.go:117] "RemoveContainer" containerID="7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5" Jul 14 19:55:06.615917 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.615899 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" event={"ID":"688a0ec9-ea1e-49db-9b49-084856d56e8d","Type":"ContainerStarted","Data":"1d86960252da58746bf670ab0d7a99fe09325eedfb574844d84f93df2a499532"} Jul 14 19:55:06.629198 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.629176 2574 scope.go:117] "RemoveContainer" containerID="7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5" Jul 14 19:55:06.629517 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:55:06.629480 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5\": container with ID starting with 7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5 not found: ID does not exist" containerID="7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5" Jul 14 19:55:06.629622 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.629522 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5"} err="failed to get container status \"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5\": rpc error: code = NotFound desc = could not find container \"7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5\": container with ID starting with 7fc51492ee66dc8eaac915908a1211cc0f74aeae43cfa1031cb120d34d7aeca5 not found: ID does not exist" Jul 14 19:55:06.643005 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.642986 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:55:06.652295 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:06.652272 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-drt88"] Jul 14 19:55:07.623395 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:07.623359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" event={"ID":"688a0ec9-ea1e-49db-9b49-084856d56e8d","Type":"ContainerStarted","Data":"cf1df93518a883e74a99390ca4ab2a7db91ee9285b3c2a62c4b420db5df8019e"} Jul 14 19:55:07.623791 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:07.623482 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:07.647068 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:07.647021 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" podStartSLOduration=1.647008204 podStartE2EDuration="1.647008204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:07.644745 +0000 UTC m=+457.489081887" watchObservedRunningTime="2026-07-14 19:55:07.647008204 +0000 UTC m=+457.491345078" Jul 14 19:55:08.600143 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:08.600109 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84408311-cbbe-42fa-8996-fa61f62bb538" path="/var/lib/kubelet/pods/84408311-cbbe-42fa-8996-fa61f62bb538/volumes" Jul 14 19:55:18.630699 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:18.630668 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 19:55:22.273881 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.273847 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6"] Jul 14 19:55:22.278344 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.278326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.280867 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.280849 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vqlq7\"" Jul 14 19:55:22.289752 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.289719 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6"] Jul 14 19:55:22.326257 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326377 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7dn\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-kube-api-access-9p7dn\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326377 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326450 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326575 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326575 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.326575 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.326568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427253 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7dn\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-kube-api-access-9p7dn\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427408 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427408 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427408 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427606 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427606 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427606 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427745 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427630 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427745 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427745 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.427943 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.428018 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.427997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.428018 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.428009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.428484 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.428458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.429779 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.429758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.429953 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.429937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.435440 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.435415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7dn\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-kube-api-access-9p7dn\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.435890 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.435869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e9ecf78-8596-49c1-90d9-e0c8498c7494-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-v6dq6\" (UID: \"6e9ecf78-8596-49c1-90d9-e0c8498c7494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.589615 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.589533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:22.728133 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.728095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6"] Jul 14 19:55:22.729052 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:55:22.729033 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9ecf78_8596_49c1_90d9_e0c8498c7494.slice/crio-8409dd116b1eb5984f3a2ca702524bebac77caa7c89c2683494c2f818f3acc34 WatchSource:0}: Error finding container 8409dd116b1eb5984f3a2ca702524bebac77caa7c89c2683494c2f818f3acc34: Status 404 returned error can't find the container with id 8409dd116b1eb5984f3a2ca702524bebac77caa7c89c2683494c2f818f3acc34 Jul 14 19:55:22.731469 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.731439 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:55:22.731548 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.731525 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:55:22.731585 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:22.731557 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:55:23.702561 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:23.702524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" event={"ID":"6e9ecf78-8596-49c1-90d9-e0c8498c7494","Type":"ContainerStarted","Data":"a08fc5ba74ff5912312a27d22f37b3f371fdc9f4eab2aaf3bfe76e18fb46fc85"} Jul 14 19:55:23.703075 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:23.702588 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" event={"ID":"6e9ecf78-8596-49c1-90d9-e0c8498c7494","Type":"ContainerStarted","Data":"8409dd116b1eb5984f3a2ca702524bebac77caa7c89c2683494c2f818f3acc34"} Jul 14 19:55:23.732287 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:23.732244 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podStartSLOduration=1.732227934 podStartE2EDuration="1.732227934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:23.731312946 +0000 UTC m=+473.575649820" watchObservedRunningTime="2026-07-14 19:55:23.732227934 +0000 UTC m=+473.576564808" Jul 14 19:55:24.590703 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:24.590662 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:24.599919 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:24.599894 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:24.708415 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:24.708382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:24.709424 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:24.709406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:55:25.367263 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.367227 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:55:25.370632 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.370611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.373119 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.373100 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h7c6z\"" Jul 14 19:55:25.373214 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.373125 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 19:55:25.378624 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.378234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:55:25.406331 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.406306 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:55:25.456399 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.456368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce3de322-d2c2-455c-8206-c8f2a611a607-config-file\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.456562 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.456438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkt8j\" (UniqueName: \"kubernetes.io/projected/ce3de322-d2c2-455c-8206-c8f2a611a607-kube-api-access-nkt8j\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.557676 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.557647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce3de322-d2c2-455c-8206-c8f2a611a607-config-file\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.557806 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.557714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkt8j\" (UniqueName: \"kubernetes.io/projected/ce3de322-d2c2-455c-8206-c8f2a611a607-kube-api-access-nkt8j\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.558265 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.558245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce3de322-d2c2-455c-8206-c8f2a611a607-config-file\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.566875 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.566842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkt8j\" (UniqueName: \"kubernetes.io/projected/ce3de322-d2c2-455c-8206-c8f2a611a607-kube-api-access-nkt8j\") pod \"limitador-limitador-54d6d988d5-82b2j\" (UID: \"ce3de322-d2c2-455c-8206-c8f2a611a607\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.682537 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.682453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:25.818901 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:25.818775 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:55:25.820988 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:55:25.820965 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3de322_d2c2_455c_8206_c8f2a611a607.slice/crio-eeb8b7f33f5fdd8d36f751d4212b4d6c20ff744cee03962afa0108bfed22196d WatchSource:0}: Error finding container eeb8b7f33f5fdd8d36f751d4212b4d6c20ff744cee03962afa0108bfed22196d: Status 404 returned error can't find the container with id eeb8b7f33f5fdd8d36f751d4212b4d6c20ff744cee03962afa0108bfed22196d Jul 14 19:55:26.722945 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:26.722909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" event={"ID":"ce3de322-d2c2-455c-8206-c8f2a611a607","Type":"ContainerStarted","Data":"eeb8b7f33f5fdd8d36f751d4212b4d6c20ff744cee03962afa0108bfed22196d"} Jul 14 19:55:28.734698 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:28.734594 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" event={"ID":"ce3de322-d2c2-455c-8206-c8f2a611a607","Type":"ContainerStarted","Data":"9675ca6f6b7db458a93b57f3cc77bfca8688b3348592ed8024e28f7e3793996c"} Jul 14 19:55:28.735075 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:28.734706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:28.753707 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:28.753659 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" podStartSLOduration=1.112657848 podStartE2EDuration="3.753643609s" totalImagesPullingTime="2.640985761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:28.750574475 +0000 UTC m=+478.594911373" watchObservedRunningTime="2026-07-14 19:55:28.753643609 +0000 UTC m=+478.597980547" Jul 14 19:55:39.740198 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:39.740165 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-82b2j" Jul 14 19:55:54.580539 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.580504 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:55:54.583632 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.583612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.587026 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.587005 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-94dvd\"" Jul 14 19:55:54.587126 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.587031 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 19:55:54.599726 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.599702 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:55:54.604566 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.604545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.604695 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.604599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6pd\" (UniqueName: \"kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.705859 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.705822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.706001 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.705890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6pd\" (UniqueName: \"kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.708530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.708475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.714660 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.714636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6pd\" (UniqueName: \"kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd\") pod \"authorino-66975df76b-rt8rf\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:54.893228 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:54.893149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:55:55.023243 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:55.023218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:55:55.024701 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:55:55.024679 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29ca57e9_e5fe_40bd_bd6d_e2956316056d.slice/crio-ab83da438b0510d49e0402a7bff69a29b54ddf23f3e933923dfc81634be39490 WatchSource:0}: Error finding container ab83da438b0510d49e0402a7bff69a29b54ddf23f3e933923dfc81634be39490: Status 404 returned error can't find the container with id ab83da438b0510d49e0402a7bff69a29b54ddf23f3e933923dfc81634be39490 Jul 14 19:55:55.865779 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:55.865734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66975df76b-rt8rf" event={"ID":"29ca57e9-e5fe-40bd-bd6d-e2956316056d","Type":"ContainerStarted","Data":"ab83da438b0510d49e0402a7bff69a29b54ddf23f3e933923dfc81634be39490"} Jul 14 19:55:58.883716 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:58.883679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66975df76b-rt8rf" event={"ID":"29ca57e9-e5fe-40bd-bd6d-e2956316056d","Type":"ContainerStarted","Data":"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee"} Jul 14 19:55:58.899530 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:55:58.899458 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66975df76b-rt8rf" podStartSLOduration=1.8514948599999999 podStartE2EDuration="4.899442333s" totalImagesPullingTime="3.047947473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:58.899073932 +0000 UTC m=+508.743410805" watchObservedRunningTime="2026-07-14 19:55:58.899442333 +0000 UTC m=+508.743779206" Jul 14 19:56:40.252529 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:40.249431 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:56:48.150260 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.150225 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerID="a08fc5ba74ff5912312a27d22f37b3f371fdc9f4eab2aaf3bfe76e18fb46fc85" exitCode=0 Jul 14 19:56:48.150673 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.150302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" event={"ID":"6e9ecf78-8596-49c1-90d9-e0c8498c7494","Type":"ContainerDied","Data":"a08fc5ba74ff5912312a27d22f37b3f371fdc9f4eab2aaf3bfe76e18fb46fc85"} Jul 14 19:56:48.150673 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.150659 2574 scope.go:117] "RemoveContainer" containerID="a08fc5ba74ff5912312a27d22f37b3f371fdc9f4eab2aaf3bfe76e18fb46fc85" Jul 14 19:56:48.151718 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.151691 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:56:48.151788 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.151775 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:56:48.151836 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:48.151822 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 19:56:49.156979 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:49.156947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" event={"ID":"6e9ecf78-8596-49c1-90d9-e0c8498c7494","Type":"ContainerStarted","Data":"aa9a10484757c5ff864a1beacd4c58e00bdac68bd4e77a8fffd166880d2a9e6a"} Jul 14 19:56:49.590370 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:49.590328 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:56:49.590606 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:49.590562 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-v6dq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:56:49.590688 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:49.590620 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podUID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 14 19:56:50.590717 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:50.590679 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-v6dq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:56:50.591106 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:50.590732 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podUID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 14 19:56:51.590691 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:51.590658 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-v6dq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:56:51.590907 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:51.590712 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podUID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 14 19:56:52.590413 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:52.590373 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-v6dq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:56:52.590603 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:52.590441 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podUID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 14 19:56:53.590000 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:53.589960 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-v6dq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 14 19:56:53.590357 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:53.590008 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" podUID="6e9ecf78-8596-49c1-90d9-e0c8498c7494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 14 19:56:54.599283 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:54.599256 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:56:54.599770 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:54.599717 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:56:54.600563 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:56:54.600540 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-v6dq6" Jul 14 19:57:07.483400 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:57:07.483362 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:57:33.162152 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:57:33.162115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:57:38.204440 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:57:38.204402 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:57:51.455432 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:57:51.455351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:58:00.767193 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:58:00.767156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:58:20.160958 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:58:20.160923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:59:18.561278 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.561249 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78d77bf8dc-6jskd"] Jul 14 19:59:18.564590 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.564571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.573330 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.573309 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78d77bf8dc-6jskd"] Jul 14 19:59:18.722470 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.722439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c02dcb30-b969-4fdc-9062-ec72db8d77ef-tls-cert\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.722649 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.722535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgb9\" (UniqueName: \"kubernetes.io/projected/c02dcb30-b969-4fdc-9062-ec72db8d77ef-kube-api-access-9mgb9\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.823261 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.823176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c02dcb30-b969-4fdc-9062-ec72db8d77ef-tls-cert\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.823261 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.823216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgb9\" (UniqueName: \"kubernetes.io/projected/c02dcb30-b969-4fdc-9062-ec72db8d77ef-kube-api-access-9mgb9\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.825898 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.825865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c02dcb30-b969-4fdc-9062-ec72db8d77ef-tls-cert\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.831621 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.831593 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgb9\" (UniqueName: \"kubernetes.io/projected/c02dcb30-b969-4fdc-9062-ec72db8d77ef-kube-api-access-9mgb9\") pod \"authorino-78d77bf8dc-6jskd\" (UID: \"c02dcb30-b969-4fdc-9062-ec72db8d77ef\") " pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:18.874011 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:18.873981 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78d77bf8dc-6jskd" Jul 14 19:59:19.030205 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.030178 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78d77bf8dc-6jskd"] Jul 14 19:59:19.032294 ip-10-0-129-28 kubenswrapper[2574]: W0714 19:59:19.032263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc02dcb30_b969_4fdc_9062_ec72db8d77ef.slice/crio-457cfe92f4af04659a27eba264f6a24cb6bc8f2beb618cc19806e76301654b54 WatchSource:0}: Error finding container 457cfe92f4af04659a27eba264f6a24cb6bc8f2beb618cc19806e76301654b54: Status 404 returned error can't find the container with id 457cfe92f4af04659a27eba264f6a24cb6bc8f2beb618cc19806e76301654b54 Jul 14 19:59:19.033681 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.033664 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:59:19.919298 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.919256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78d77bf8dc-6jskd" event={"ID":"c02dcb30-b969-4fdc-9062-ec72db8d77ef","Type":"ContainerStarted","Data":"8917b3634c7af03809761dadf0aa9f670967e7d182de9d9ed524485fbf121a06"} Jul 14 19:59:19.919298 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.919311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78d77bf8dc-6jskd" event={"ID":"c02dcb30-b969-4fdc-9062-ec72db8d77ef","Type":"ContainerStarted","Data":"457cfe92f4af04659a27eba264f6a24cb6bc8f2beb618cc19806e76301654b54"} Jul 14 19:59:19.938430 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.938370 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78d77bf8dc-6jskd" podStartSLOduration=1.574966071 podStartE2EDuration="1.938354571s" totalImagesPullingTime="363.3885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:59:19.93691388 +0000 UTC m=+709.781250746" watchObservedRunningTime="2026-07-14 19:59:19.938354571 +0000 UTC m=+709.782691445" Jul 14 19:59:19.970288 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.970252 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:59:19.970464 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:19.970440 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-66975df76b-rt8rf" podUID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" containerName="authorino" containerID="cri-o://432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee" gracePeriod=30 Jul 14 19:59:20.230393 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.230368 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:59:20.334788 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.334748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6pd\" (UniqueName: \"kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd\") pod \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " Jul 14 19:59:20.334980 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.334847 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert\") pod \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\" (UID: \"29ca57e9-e5fe-40bd-bd6d-e2956316056d\") " Jul 14 19:59:20.337012 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.336976 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd" (OuterVolumeSpecName: "kube-api-access-8b6pd") pod "29ca57e9-e5fe-40bd-bd6d-e2956316056d" (UID: "29ca57e9-e5fe-40bd-bd6d-e2956316056d"). InnerVolumeSpecName "kube-api-access-8b6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:59:20.345142 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.345119 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "29ca57e9-e5fe-40bd-bd6d-e2956316056d" (UID: "29ca57e9-e5fe-40bd-bd6d-e2956316056d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:59:20.436620 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.436543 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6pd\" (UniqueName: \"kubernetes.io/projected/29ca57e9-e5fe-40bd-bd6d-e2956316056d-kube-api-access-8b6pd\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:59:20.436620 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.436577 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29ca57e9-e5fe-40bd-bd6d-e2956316056d-tls-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 19:59:20.929208 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.929120 2574 generic.go:356] "Generic (PLEG): container finished" podID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" containerID="432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee" exitCode=0 Jul 14 19:59:20.929208 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.929195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66975df76b-rt8rf" event={"ID":"29ca57e9-e5fe-40bd-bd6d-e2956316056d","Type":"ContainerDied","Data":"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee"} Jul 14 19:59:20.929709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.929240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66975df76b-rt8rf" event={"ID":"29ca57e9-e5fe-40bd-bd6d-e2956316056d","Type":"ContainerDied","Data":"ab83da438b0510d49e0402a7bff69a29b54ddf23f3e933923dfc81634be39490"} Jul 14 19:59:20.929709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.929263 2574 scope.go:117] "RemoveContainer" containerID="432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee" Jul 14 19:59:20.929709 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.929408 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66975df76b-rt8rf" Jul 14 19:59:20.943234 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.943203 2574 scope.go:117] "RemoveContainer" containerID="432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee" Jul 14 19:59:20.943570 ip-10-0-129-28 kubenswrapper[2574]: E0714 19:59:20.943520 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee\": container with ID starting with 432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee not found: ID does not exist" containerID="432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee" Jul 14 19:59:20.943570 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.943555 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee"} err="failed to get container status \"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee\": rpc error: code = NotFound desc = could not find container \"432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee\": container with ID starting with 432d279100800db090663776d407cfbe6c3fc3b4b4e385beff5d65a1b55643ee not found: ID does not exist" Jul 14 19:59:20.953119 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.953095 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:59:20.957364 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:20.957339 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66975df76b-rt8rf"] Jul 14 19:59:22.606730 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:22.606657 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" path="/var/lib/kubelet/pods/29ca57e9-e5fe-40bd-bd6d-e2956316056d/volumes" Jul 14 19:59:47.356792 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:47.356757 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 19:59:57.360933 ip-10-0-129-28 kubenswrapper[2574]: I0714 19:59:57.360897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:00:08.168243 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:00:08.168203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:00:18.065455 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:00:18.065413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:00:28.658986 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:00:28.658955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:00:57.963205 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:00:57.963123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:01:10.767549 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:01:10.767512 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:01:44.861953 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:01:44.861921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:01:54.256050 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:01:54.256016 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:02.658513 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:02.658458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:12.054671 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:12.054635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:22.864668 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:22.864571 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:34.767277 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:34.767238 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:45.851866 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:45.851822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:02:56.161675 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:02:56.161643 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:03:21.164593 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:03:21.164558 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:03:30.253458 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:03:30.253425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:03:53.058041 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:03:53.057961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:03.054627 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:03.054583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:15.164796 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:15.164758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:23.262310 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:23.262271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:33.169428 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:33.169391 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:42.755616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:42.755574 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:04:52.357872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:04:52.357831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:05:01.868133 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:01.868095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:05:27.618697 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:27.618596 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 20:05:27.619282 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:27.618865 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" podUID="688a0ec9-ea1e-49db-9b49-084856d56e8d" containerName="manager" containerID="cri-o://cf1df93518a883e74a99390ca4ab2a7db91ee9285b3c2a62c4b420db5df8019e" gracePeriod=10 Jul 14 20:05:30.566563 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.566539 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 20:05:30.598439 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.598409 2574 scope.go:117] "RemoveContainer" containerID="cf1df93518a883e74a99390ca4ab2a7db91ee9285b3c2a62c4b420db5df8019e" Jul 14 20:05:30.697690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.697664 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume\") pod \"688a0ec9-ea1e-49db-9b49-084856d56e8d\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " Jul 14 20:05:30.697797 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.697707 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfmvw\" (UniqueName: \"kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw\") pod \"688a0ec9-ea1e-49db-9b49-084856d56e8d\" (UID: \"688a0ec9-ea1e-49db-9b49-084856d56e8d\") " Jul 14 20:05:30.698086 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.698059 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "688a0ec9-ea1e-49db-9b49-084856d56e8d" (UID: "688a0ec9-ea1e-49db-9b49-084856d56e8d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:05:30.699815 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.699794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw" (OuterVolumeSpecName: "kube-api-access-kfmvw") pod "688a0ec9-ea1e-49db-9b49-084856d56e8d" (UID: "688a0ec9-ea1e-49db-9b49-084856d56e8d"). InnerVolumeSpecName "kube-api-access-kfmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:05:30.798234 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.798211 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/688a0ec9-ea1e-49db-9b49-084856d56e8d-extensions-socket-volume\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:05:30.798380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.798237 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmvw\" (UniqueName: \"kubernetes.io/projected/688a0ec9-ea1e-49db-9b49-084856d56e8d-kube-api-access-kfmvw\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:05:30.825619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.825582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" event={"ID":"688a0ec9-ea1e-49db-9b49-084856d56e8d","Type":"ContainerDied","Data":"cf1df93518a883e74a99390ca4ab2a7db91ee9285b3c2a62c4b420db5df8019e"} Jul 14 20:05:30.825737 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.825623 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" Jul 14 20:05:30.825737 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.825642 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497" event={"ID":"688a0ec9-ea1e-49db-9b49-084856d56e8d","Type":"ContainerDied","Data":"1d86960252da58746bf670ab0d7a99fe09325eedfb574844d84f93df2a499532"} Jul 14 20:05:30.859112 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.859090 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 20:05:30.863517 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:30.863481 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nz497"] Jul 14 20:05:32.604717 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:05:32.604673 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688a0ec9-ea1e-49db-9b49-084856d56e8d" path="/var/lib/kubelet/pods/688a0ec9-ea1e-49db-9b49-084856d56e8d/volumes" Jul 14 20:06:28.846058 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846023 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4"] Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846363 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688a0ec9-ea1e-49db-9b49-084856d56e8d" containerName="manager" Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846375 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="688a0ec9-ea1e-49db-9b49-084856d56e8d" containerName="manager" Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846387 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" containerName="authorino" Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846394 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" containerName="authorino" Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846457 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="688a0ec9-ea1e-49db-9b49-084856d56e8d" containerName="manager" Jul 14 20:06:28.846541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.846465 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="29ca57e9-e5fe-40bd-bd6d-e2956316056d" containerName="authorino" Jul 14 20:06:28.849422 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.849406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:28.852276 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.852259 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-65fsx\"" Jul 14 20:06:28.861630 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.861607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4"] Jul 14 20:06:28.961411 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.961373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda53b00-ca4e-4984-a2d9-c902960fcab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:28.961578 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:28.961444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m885n\" (UniqueName: \"kubernetes.io/projected/eda53b00-ca4e-4984-a2d9-c902960fcab7-kube-api-access-m885n\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.062441 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.062406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda53b00-ca4e-4984-a2d9-c902960fcab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.062580 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.062473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m885n\" (UniqueName: \"kubernetes.io/projected/eda53b00-ca4e-4984-a2d9-c902960fcab7-kube-api-access-m885n\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.062807 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.062783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eda53b00-ca4e-4984-a2d9-c902960fcab7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.076267 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.076243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m885n\" (UniqueName: \"kubernetes.io/projected/eda53b00-ca4e-4984-a2d9-c902960fcab7-kube-api-access-m885n\") pod \"kuadrant-operator-controller-manager-86878c769f-whsp4\" (UID: \"eda53b00-ca4e-4984-a2d9-c902960fcab7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.159093 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.159026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:29.298032 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:29.298009 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4"] Jul 14 20:06:29.299945 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:06:29.299918 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda53b00_ca4e_4984_a2d9_c902960fcab7.slice/crio-7a6bf1b76783192768f30b1aec9139e9a8d7fabd20879425eb341b534c5fb92f WatchSource:0}: Error finding container 7a6bf1b76783192768f30b1aec9139e9a8d7fabd20879425eb341b534c5fb92f: Status 404 returned error can't find the container with id 7a6bf1b76783192768f30b1aec9139e9a8d7fabd20879425eb341b534c5fb92f Jul 14 20:06:30.111385 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:30.111350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" event={"ID":"eda53b00-ca4e-4984-a2d9-c902960fcab7","Type":"ContainerStarted","Data":"50c33afa7eb971a623c9724ed589791c1ac7795b1235777441d09a25879f39ff"} Jul 14 20:06:30.111813 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:30.111407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" event={"ID":"eda53b00-ca4e-4984-a2d9-c902960fcab7","Type":"ContainerStarted","Data":"7a6bf1b76783192768f30b1aec9139e9a8d7fabd20879425eb341b534c5fb92f"} Jul 14 20:06:30.111813 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:30.111437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:30.136913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:30.136864 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" podStartSLOduration=2.136848295 podStartE2EDuration="2.136848295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:06:30.134391373 +0000 UTC m=+1139.978728290" watchObservedRunningTime="2026-07-14 20:06:30.136848295 +0000 UTC m=+1139.981185245" Jul 14 20:06:41.119556 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:41.119519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-whsp4" Jul 14 20:06:50.764122 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:06:50.764083 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:02.559278 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:02.559246 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:11.169034 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:11.169000 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:21.862476 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:21.862437 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:32.367009 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:32.366974 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:43.060347 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:43.060311 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:07:52.161237 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:07:52.161201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:08:05.358932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:08:05.358893 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:08:14.756218 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:08:14.756182 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:08:23.656978 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:08:23.656901 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:08:34.156705 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:08:34.156671 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:08:55.863325 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:08:55.863291 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:09:05.358221 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:09:05.358178 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:09:15.359410 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:09:15.359370 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:09:33.666240 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:09:33.666201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:09:45.166913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:09:45.166872 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:09:54.958473 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:09:54.958395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:03.964142 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:03.964104 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:11.865159 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:11.865115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:21.857126 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:21.857088 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:31.457336 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:31.457295 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:44.863349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:44.863313 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:10:54.960200 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:10:54.960157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:04.572722 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:04.572691 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:14.364854 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:14.364816 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:26.852287 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:26.852193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:34.375567 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:34.375531 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:46.761942 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:46.761907 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:11:56.162603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:11:56.162559 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:12:06.853799 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:06.853767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:12:14.351561 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:14.351522 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:12:27.656485 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:27.656445 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:12:38.458539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:38.458501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:12:45.721401 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.721369 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:12:45.725143 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.725123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.727671 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.727651 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84a35a9d-openshift-default-dockercfg-6gsqn\"" Jul 14 20:12:45.741577 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.741548 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:12:45.794321 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794442 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794442 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794442 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794641 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794641 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794641 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794771 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.794771 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.794712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bvf\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.895964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.895921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.895964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.895968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896210 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.895997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bvf\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896210 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896210 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896357 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896357 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896357 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896357 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896670 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896764 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.896810 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.896756 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.897235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.897211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.898806 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.898771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.898943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.898921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.904287 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.904256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:45.904833 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:45.904811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bvf\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf\") pod \"e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:46.036439 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:46.036364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:46.174362 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:46.174334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:12:46.175923 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:12:46.175892 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b553d7c_62cc_491a_91de_96eabe4f261e.slice/crio-945a38f082b8da9147506146c50bde572825e05c0736e7c71be2d19fcdd8540d WatchSource:0}: Error finding container 945a38f082b8da9147506146c50bde572825e05c0736e7c71be2d19fcdd8540d: Status 404 returned error can't find the container with id 945a38f082b8da9147506146c50bde572825e05c0736e7c71be2d19fcdd8540d Jul 14 20:12:46.178133 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:46.178100 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:12:46.178207 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:46.178174 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:12:46.178242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:46.178208 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:12:47.042602 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:47.042551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerStarted","Data":"717944e707d8a421b3f1fcc00e69e4c7c3cfc520cec0662436f03f7d0d4ffe9a"} Jul 14 20:12:47.043102 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:47.042673 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerStarted","Data":"945a38f082b8da9147506146c50bde572825e05c0736e7c71be2d19fcdd8540d"} Jul 14 20:12:47.072726 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:47.072676 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" podStartSLOduration=2.072661094 podStartE2EDuration="2.072661094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:47.070228373 +0000 UTC m=+1516.914565271" watchObservedRunningTime="2026-07-14 20:12:47.072661094 +0000 UTC m=+1516.916997967" Jul 14 20:12:48.037443 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:48.037407 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:48.042756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:48.042728 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:48.047828 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:48.047806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:48.048573 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:48.048553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:12:53.554522 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:12:53.554429 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:13:02.124862 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.124822 2574 generic.go:356] "Generic (PLEG): container finished" podID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerID="717944e707d8a421b3f1fcc00e69e4c7c3cfc520cec0662436f03f7d0d4ffe9a" exitCode=0 Jul 14 20:13:02.125368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.124887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerDied","Data":"717944e707d8a421b3f1fcc00e69e4c7c3cfc520cec0662436f03f7d0d4ffe9a"} Jul 14 20:13:02.125368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.125353 2574 scope.go:117] "RemoveContainer" containerID="717944e707d8a421b3f1fcc00e69e4c7c3cfc520cec0662436f03f7d0d4ffe9a" Jul 14 20:13:02.126387 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.126360 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:13:02.126446 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.126426 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:13:02.126480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:02.126456 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:13:03.047913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:03.047864 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:13:03.132464 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:03.132423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerStarted","Data":"29d15cd173c9426fcf8441bc9a81648fcdee983c0f2ad04fc7c784bcc9a9fcd6"} Jul 14 20:13:04.036736 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:04.036698 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:13:04.038015 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:04.037983 2574 patch_prober.go:28] interesting pod/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 20:13:04.038150 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:04.038042 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 20:13:05.037383 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:05.037341 2574 patch_prober.go:28] interesting pod/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 20:13:05.037884 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:05.037419 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 20:13:06.036812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:06.036776 2574 patch_prober.go:28] interesting pod/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 20:13:06.036986 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:06.036842 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 20:13:07.040717 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:07.040687 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:13:07.041073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:07.040951 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:13:07.041728 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:07.041710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:13:55.449354 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:13:55.449319 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:14:07.333137 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.333098 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:14:07.333705 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.333366 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" containerID="cri-o://29d15cd173c9426fcf8441bc9a81648fcdee983c0f2ad04fc7c784bcc9a9fcd6" gracePeriod=30 Jul 14 20:14:07.911945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.911912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:14:07.915449 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.915431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.918665 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.918646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9ead5203-gw-openshift-default-dockercfg-j9jqh\"" Jul 14 20:14:07.927468 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.927446 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:14:07.990009 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.989984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990118 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990118 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990046 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990118 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990140 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgxs\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:07.990422 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:07.990263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091119 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgxs\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091830 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091629 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091830 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.091953 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.092176 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.091988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.093679 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.093654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.093840 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.093815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.101420 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.101389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.102010 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.101993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgxs\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs\") pod \"e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.226102 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.226070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:08.363027 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.363003 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:14:08.364904 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:14:08.364873 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49090782_bb20_4424_a4fd_5bae88809d3a.slice/crio-0259eac83606dd1f45edf29b4d816c0fec3c72ce3544be14ca2d7abadfdb4db7 WatchSource:0}: Error finding container 0259eac83606dd1f45edf29b4d816c0fec3c72ce3544be14ca2d7abadfdb4db7: Status 404 returned error can't find the container with id 0259eac83606dd1f45edf29b4d816c0fec3c72ce3544be14ca2d7abadfdb4db7 Jul 14 20:14:08.367037 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.367007 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:08.367135 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.367070 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:08.367135 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.367099 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:08.484774 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.484689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerStarted","Data":"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213"} Jul 14 20:14:08.484774 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.484755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerStarted","Data":"0259eac83606dd1f45edf29b4d816c0fec3c72ce3544be14ca2d7abadfdb4db7"} Jul 14 20:14:08.505736 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:08.505364 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podStartSLOduration=1.505345181 podStartE2EDuration="1.505345181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:14:08.502872377 +0000 UTC m=+1598.347209252" watchObservedRunningTime="2026-07-14 20:14:08.505345181 +0000 UTC m=+1598.349682057" Jul 14 20:14:09.226874 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:09.226834 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:09.232157 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:09.232131 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:09.490059 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:09.489971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:09.491058 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:09.491040 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:11.364019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:11.363983 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:14:12.542006 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.541967 2574 generic.go:356] "Generic (PLEG): container finished" podID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerID="29d15cd173c9426fcf8441bc9a81648fcdee983c0f2ad04fc7c784bcc9a9fcd6" exitCode=0 Jul 14 20:14:12.542473 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.542025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerDied","Data":"29d15cd173c9426fcf8441bc9a81648fcdee983c0f2ad04fc7c784bcc9a9fcd6"} Jul 14 20:14:12.542473 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.542087 2574 scope.go:117] "RemoveContainer" containerID="717944e707d8a421b3f1fcc00e69e4c7c3cfc520cec0662436f03f7d0d4ffe9a" Jul 14 20:14:12.620839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.620694 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:14:12.737360 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737324 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737504 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737504 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737589 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737555 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737632 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737601 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5bvf\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737673 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737718 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737678 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737718 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737713 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737819 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:14:12.737819 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737747 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket\") pod \"0b553d7c-62cc-491a-91de-96eabe4f261e\" (UID: \"0b553d7c-62cc-491a-91de-96eabe4f261e\") " Jul 14 20:14:12.737919 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.737883 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:14:12.738145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.738085 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.738145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.738108 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b553d7c-62cc-491a-91de-96eabe4f261e-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.738325 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.738162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:14:12.738325 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.738190 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:14:12.738325 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.738248 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:14:12.740275 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.740253 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:14:12.740788 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.740763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf" (OuterVolumeSpecName: "kube-api-access-n5bvf") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "kube-api-access-n5bvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:14:12.740788 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.740768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:14:12.742042 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.742020 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b553d7c-62cc-491a-91de-96eabe4f261e" (UID: "0b553d7c-62cc-491a-91de-96eabe4f261e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:14:12.838596 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838554 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838596 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838588 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bvf\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-kube-api-access-n5bvf\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838596 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838602 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838615 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838625 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838637 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b553d7c-62cc-491a-91de-96eabe4f261e-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:12.838839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:12.838647 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b553d7c-62cc-491a-91de-96eabe4f261e-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:14:13.548187 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:13.548152 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" event={"ID":"0b553d7c-62cc-491a-91de-96eabe4f261e","Type":"ContainerDied","Data":"945a38f082b8da9147506146c50bde572825e05c0736e7c71be2d19fcdd8540d"} Jul 14 20:14:13.548187 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:13.548186 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq" Jul 14 20:14:13.548690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:13.548210 2574 scope.go:117] "RemoveContainer" containerID="29d15cd173c9426fcf8441bc9a81648fcdee983c0f2ad04fc7c784bcc9a9fcd6" Jul 14 20:14:13.571964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:13.571942 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:14:13.577311 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:13.577284 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84a35a9d-openshift-default-6ddd6678df-vzscq"] Jul 14 20:14:14.600166 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:14.600132 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" path="/var/lib/kubelet/pods/0b553d7c-62cc-491a-91de-96eabe4f261e/volumes" Jul 14 20:14:19.580420 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.580386 2574 generic.go:356] "Generic (PLEG): container finished" podID="49090782-bb20-4424-a4fd-5bae88809d3a" containerID="2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213" exitCode=0 Jul 14 20:14:19.580859 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.580462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerDied","Data":"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213"} Jul 14 20:14:19.580928 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.580913 2574 scope.go:117] "RemoveContainer" containerID="2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213" Jul 14 20:14:19.581863 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.581838 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:19.581928 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.581900 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:19.581967 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:19.581931 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:14:20.599142 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:20.599102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerStarted","Data":"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b"} Jul 14 20:14:21.226938 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:21.226720 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:21.228180 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:21.227761 2574 patch_prober.go:28] interesting pod/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:14:21.228180 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:21.227815 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:14:22.227019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:22.226972 2574 patch_prober.go:28] interesting pod/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:14:22.227541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:22.227044 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:14:23.227620 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:23.227578 2574 patch_prober.go:28] interesting pod/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:14:23.228017 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:23.227662 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:14:24.227461 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:24.227423 2574 patch_prober.go:28] interesting pod/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 20:14:24.227685 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:24.227526 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 20:14:24.490935 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:24.490841 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:25.231557 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:25.231529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:14:25.232568 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:14:25.232545 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:15:13.885018 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:13.884982 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:15:13.885577 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:13.885237 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" containerID="cri-o://9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b" gracePeriod=30 Jul 14 20:15:16.653041 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:16.653005 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:15:19.145448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.145425 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:15:19.198615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198588 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.198721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198632 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.198721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198667 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.198721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198700 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.198867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.198971 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198945 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.199062 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.198959 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:19.199062 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199021 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wgxs\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.199062 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199059 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.199213 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199104 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs\") pod \"49090782-bb20-4424-a4fd-5bae88809d3a\" (UID: \"49090782-bb20-4424-a4fd-5bae88809d3a\") " Jul 14 20:15:19.199213 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:15:19.199756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199408 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.199756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199427 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49090782-bb20-4424-a4fd-5bae88809d3a-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.199756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:19.199756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.199626 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:19.200140 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.200110 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:19.201760 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.201725 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:15:19.201858 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.201819 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:15:19.201858 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.201827 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:15:19.201962 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.201883 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs" (OuterVolumeSpecName: "kube-api-access-6wgxs") pod "49090782-bb20-4424-a4fd-5bae88809d3a" (UID: "49090782-bb20-4424-a4fd-5bae88809d3a"). InnerVolumeSpecName "kube-api-access-6wgxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:15:19.300447 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300404 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300447 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300445 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300447 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300453 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49090782-bb20-4424-a4fd-5bae88809d3a-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300447 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300462 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300760 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300470 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgxs\" (UniqueName: \"kubernetes.io/projected/49090782-bb20-4424-a4fd-5bae88809d3a-kube-api-access-6wgxs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300760 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300479 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.300760 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.300506 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49090782-bb20-4424-a4fd-5bae88809d3a-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:15:19.906389 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.906351 2574 generic.go:356] "Generic (PLEG): container finished" podID="49090782-bb20-4424-a4fd-5bae88809d3a" containerID="9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b" exitCode=0 Jul 14 20:15:19.906616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.906416 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" Jul 14 20:15:19.906616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.906419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerDied","Data":"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b"} Jul 14 20:15:19.906616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.906529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs" event={"ID":"49090782-bb20-4424-a4fd-5bae88809d3a","Type":"ContainerDied","Data":"0259eac83606dd1f45edf29b4d816c0fec3c72ce3544be14ca2d7abadfdb4db7"} Jul 14 20:15:19.906616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.906563 2574 scope.go:117] "RemoveContainer" containerID="9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b" Jul 14 20:15:19.916793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.916774 2574 scope.go:117] "RemoveContainer" containerID="2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213" Jul 14 20:15:19.924898 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.924877 2574 scope.go:117] "RemoveContainer" containerID="9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b" Jul 14 20:15:19.925170 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:15:19.925151 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b\": container with ID starting with 9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b not found: ID does not exist" containerID="9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b" Jul 14 20:15:19.925217 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.925179 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b"} err="failed to get container status \"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b\": rpc error: code = NotFound desc = could not find container \"9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b\": container with ID starting with 9e1ae92965260b448ce56d52fa40194408811e8152cff3d08345e723d258fb2b not found: ID does not exist" Jul 14 20:15:19.925217 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.925196 2574 scope.go:117] "RemoveContainer" containerID="2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213" Jul 14 20:15:19.925450 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:15:19.925432 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213\": container with ID starting with 2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213 not found: ID does not exist" containerID="2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213" Jul 14 20:15:19.925539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.925456 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213"} err="failed to get container status \"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213\": rpc error: code = NotFound desc = could not find container \"2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213\": container with ID starting with 2449c3b26ebdc49517381c1c7f93a6a17ef94942e81c1b8dd1c19e5f061bf213 not found: ID does not exist" Jul 14 20:15:19.932086 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.932056 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:15:19.938932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:19.938903 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9ead5203-gw-openshift-default-5865f954f9-qn2xs"] Jul 14 20:15:20.599435 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.599395 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" path="/var/lib/kubelet/pods/49090782-bb20-4424-a4fd-5bae88809d3a/volumes" Jul 14 20:15:20.652051 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652027 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:15:20.652356 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652344 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652358 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652368 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652373 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652382 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652388 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.652396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652396 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:15:20.652612 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652402 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:15:20.652612 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652459 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.652612 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652467 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:15:20.652612 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.652476 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b553d7c-62cc-491a-91de-96eabe4f261e" containerName="istio-proxy" Jul 14 20:15:20.656749 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.656731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.660092 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.660073 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-96fd5154-openshift-default-dockercfg-t645v\"" Jul 14 20:15:20.675847 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.675826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:15:20.713351 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713467 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713467 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713467 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8wk\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713610 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713456 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713610 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713610 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713610 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.713753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.713621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.814944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.814908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.814944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.814942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815162 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.814958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815162 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815162 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815162 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815371 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8wk\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815371 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815371 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815555 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.815920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.816031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815974 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.816031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.816031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.815939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.817439 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.817420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.818116 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.818100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.827082 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.827058 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.827211 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.827062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8wk\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk\") pod \"e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:20.966304 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:20.966263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:21.111825 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.111800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:15:21.113599 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:15:21.113571 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3554140c_3e66_4412_b0c0_cff892363c6c.slice/crio-35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3 WatchSource:0}: Error finding container 35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3: Status 404 returned error can't find the container with id 35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3 Jul 14 20:15:21.115803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.115769 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:21.115895 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.115839 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:21.115895 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.115868 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:21.922735 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.922698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerStarted","Data":"fe48aa22746e9f5a112bc03b0b8e199a253ad9eccd96d05afc68c37dd532e45f"} Jul 14 20:15:21.923129 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.922749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerStarted","Data":"35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3"} Jul 14 20:15:21.946793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.946741 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podStartSLOduration=1.946721683 podStartE2EDuration="1.946721683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:15:21.946049637 +0000 UTC m=+1671.790386542" watchObservedRunningTime="2026-07-14 20:15:21.946721683 +0000 UTC m=+1671.791058556" Jul 14 20:15:21.967394 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:21.967368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:22.971092 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:22.971058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:23.933886 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:23.933860 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:23.934917 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:23.934894 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:24.359539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:24.359501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:15:32.984807 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.984765 2574 generic.go:356] "Generic (PLEG): container finished" podID="3554140c-3e66-4412-b0c0-cff892363c6c" containerID="fe48aa22746e9f5a112bc03b0b8e199a253ad9eccd96d05afc68c37dd532e45f" exitCode=0 Jul 14 20:15:32.985233 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.984840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerDied","Data":"fe48aa22746e9f5a112bc03b0b8e199a253ad9eccd96d05afc68c37dd532e45f"} Jul 14 20:15:32.985233 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.985199 2574 scope.go:117] "RemoveContainer" containerID="fe48aa22746e9f5a112bc03b0b8e199a253ad9eccd96d05afc68c37dd532e45f" Jul 14 20:15:32.986201 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.986176 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:32.986275 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.986242 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:32.986314 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:32.986280 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:15:33.992721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:33.992684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerStarted","Data":"a4d3ccbd19387541df32bd9358a9acaa964336e8e693336ff65579e423d262ce"} Jul 14 20:15:34.966686 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:34.966641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:34.967885 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:34.967857 2574 patch_prober.go:28] interesting pod/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 20:15:34.967998 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:34.967925 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 20:15:35.966638 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:35.966601 2574 patch_prober.go:28] interesting pod/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 20:15:35.967106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:35.966676 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 20:15:36.967349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:36.967318 2574 patch_prober.go:28] interesting pod/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 20:15:36.967756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:36.967374 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 20:15:37.966819 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:37.966781 2574 patch_prober.go:28] interesting pod/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 20:15:37.966998 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:37.966836 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 20:15:38.934231 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:38.934197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:38.970483 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:38.970460 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:15:39.025828 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:15:39.025804 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:16:30.258085 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:30.258047 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:16:43.095441 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.095403 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:16:43.096006 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.095697 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" containerID="cri-o://a4d3ccbd19387541df32bd9358a9acaa964336e8e693336ff65579e423d262ce" gracePeriod=30 Jul 14 20:16:43.405118 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.405038 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:16:43.405683 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.405654 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49090782-bb20-4424-a4fd-5bae88809d3a" containerName="istio-proxy" Jul 14 20:16:43.409013 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.408992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.411584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.411560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f02785bb-openshift-default-dockercfg-rhbm2\"" Jul 14 20:16:43.423067 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.423045 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:16:43.503941 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.503912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504055 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.503977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t627f\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504055 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.503998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504055 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504055 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504229 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504229 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504229 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.504375 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.504272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605681 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605681 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t627f\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.605932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606105 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606105 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.605969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606105 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606534 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.606692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.607063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.606798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.608210 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.608191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.608942 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.608922 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.614973 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.614951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.615056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.615041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t627f\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f\") pod \"e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.724378 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.724350 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:43.861051 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.861011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:16:43.862135 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:16:43.862110 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadcc031_8353_4a3f_a32e_44735c0adb3a.slice/crio-25baa5245482cd432f694226afd8dfc45033fe3ade21cbc6e3d35dd2996f92c7 WatchSource:0}: Error finding container 25baa5245482cd432f694226afd8dfc45033fe3ade21cbc6e3d35dd2996f92c7: Status 404 returned error can't find the container with id 25baa5245482cd432f694226afd8dfc45033fe3ade21cbc6e3d35dd2996f92c7 Jul 14 20:16:43.864443 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.864410 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:43.864543 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.864476 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:43.864543 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:43.864529 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:44.353657 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:44.353620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerStarted","Data":"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3"} Jul 14 20:16:44.354057 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:44.353682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerStarted","Data":"25baa5245482cd432f694226afd8dfc45033fe3ade21cbc6e3d35dd2996f92c7"} Jul 14 20:16:44.378576 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:44.378513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podStartSLOduration=1.378474966 podStartE2EDuration="1.378474966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:16:44.375019971 +0000 UTC m=+1754.219356846" watchObservedRunningTime="2026-07-14 20:16:44.378474966 +0000 UTC m=+1754.222811841" Jul 14 20:16:44.725221 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:44.725185 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:44.729922 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:44.729891 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:45.358544 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:45.358516 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:45.359528 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:45.359506 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:47.254573 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:47.254536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:16:48.376126 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.376094 2574 generic.go:356] "Generic (PLEG): container finished" podID="3554140c-3e66-4412-b0c0-cff892363c6c" containerID="a4d3ccbd19387541df32bd9358a9acaa964336e8e693336ff65579e423d262ce" exitCode=0 Jul 14 20:16:48.376565 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.376122 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerDied","Data":"a4d3ccbd19387541df32bd9358a9acaa964336e8e693336ff65579e423d262ce"} Jul 14 20:16:48.376565 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.376178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" event={"ID":"3554140c-3e66-4412-b0c0-cff892363c6c","Type":"ContainerDied","Data":"35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3"} Jul 14 20:16:48.376565 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.376197 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35d4dca8117ebb1b73ef91633e14cc22640d338b9f47613b3cb0e268396171d3" Jul 14 20:16:48.376565 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.376210 2574 scope.go:117] "RemoveContainer" containerID="fe48aa22746e9f5a112bc03b0b8e199a253ad9eccd96d05afc68c37dd532e45f" Jul 14 20:16:48.385796 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.385779 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:16:48.552456 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552417 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552456 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552588 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l8wk\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552606 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552630 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.552724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552689 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy\") pod \"3554140c-3e66-4412-b0c0-cff892363c6c\" (UID: \"3554140c-3e66-4412-b0c0-cff892363c6c\") " Jul 14 20:16:48.553043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:48.553043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552928 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:48.553043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552939 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:48.553043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.552998 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.553043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.553028 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:16:48.553426 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.553401 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:48.554991 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.554967 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk" (OuterVolumeSpecName: "kube-api-access-2l8wk") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "kube-api-access-2l8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:16:48.555085 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.554986 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:16:48.555470 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.555454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:16:48.555538 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.555474 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3554140c-3e66-4412-b0c0-cff892363c6c" (UID: "3554140c-3e66-4412-b0c0-cff892363c6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654310 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654339 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654348 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3554140c-3e66-4412-b0c0-cff892363c6c-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654356 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654364 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8wk\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-kube-api-access-2l8wk\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654372 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3554140c-3e66-4412-b0c0-cff892363c6c-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654380 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3554140c-3e66-4412-b0c0-cff892363c6c-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:48.654405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:48.654387 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3554140c-3e66-4412-b0c0-cff892363c6c-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:16:49.382539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:49.382484 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb" Jul 14 20:16:49.417257 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:49.417224 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:16:49.422218 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:49.422194 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-96fd5154-openshift-default-757c5f574c-cg8nb"] Jul 14 20:16:50.600014 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:50.599982 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" path="/var/lib/kubelet/pods/3554140c-3e66-4412-b0c0-cff892363c6c/volumes" Jul 14 20:16:55.417840 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.417742 2574 generic.go:356] "Generic (PLEG): container finished" podID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerID="8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3" exitCode=0 Jul 14 20:16:55.417840 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.417820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerDied","Data":"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3"} Jul 14 20:16:55.418386 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.418339 2574 scope.go:117] "RemoveContainer" containerID="8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3" Jul 14 20:16:55.419364 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.419341 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:55.419423 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.419408 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:55.419483 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:55.419469 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:16:56.430087 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:56.429054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerStarted","Data":"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae"} Jul 14 20:16:56.725845 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:56.725817 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:16:56.726917 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:56.726884 2574 patch_prober.go:28] interesting pod/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 20:16:56.727052 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:56.726934 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 20:16:57.725283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:57.725238 2574 patch_prober.go:28] interesting pod/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 20:16:57.725804 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:57.725313 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 20:16:58.725096 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:58.725051 2574 patch_prober.go:28] interesting pod/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 20:16:58.725301 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:58.725117 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 20:16:59.725602 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:59.725555 2574 patch_prober.go:28] interesting pod/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 20:16:59.725985 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:16:59.725623 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 20:17:00.359325 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:17:00.359290 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:17:00.728564 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:17:00.728533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:17:00.729457 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:17:00.729442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:18:02.457604 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:02.457555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:18:14.287216 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:14.287177 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:18:14.287766 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:14.287384 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" containerID="cri-o://6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae" gracePeriod=30 Jul 14 20:18:15.731440 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:15.731388 2574 patch_prober.go:28] interesting pod/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:18:15.731864 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:15.731521 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:18:19.543556 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.543529 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:18:19.655922 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.655887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.655931 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.655960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656018 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656190 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656073 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656190 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t627f\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656189 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656223 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656242 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:18:19.656299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data\") pod \"dadcc031-8353-4a3f-a32e-44735c0adb3a\" (UID: \"dadcc031-8353-4a3f-a32e-44735c0adb3a\") " Jul 14 20:18:19.656541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656350 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:18:19.656541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656366 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:18:19.656675 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656649 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:18:19.656675 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656660 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dadcc031-8353-4a3f-a32e-44735c0adb3a-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.656801 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656698 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.656857 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.656775 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.657387 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.657358 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:18:19.658359 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.658329 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:18:19.658359 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.658337 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:18:19.658555 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.658414 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f" (OuterVolumeSpecName: "kube-api-access-t627f") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "kube-api-access-t627f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:18:19.658601 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.658583 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dadcc031-8353-4a3f-a32e-44735c0adb3a" (UID: "dadcc031-8353-4a3f-a32e-44735c0adb3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:18:19.757581 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757548 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t627f\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-kube-api-access-t627f\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.757581 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757573 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.757581 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757583 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.757581 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757591 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.757932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757601 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.757932 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.757609 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dadcc031-8353-4a3f-a32e-44735c0adb3a-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:18:19.853463 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.853372 2574 generic.go:356] "Generic (PLEG): container finished" podID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerID="6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae" exitCode=0 Jul 14 20:18:19.853463 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.853420 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" Jul 14 20:18:19.853690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.853455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerDied","Data":"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae"} Jul 14 20:18:19.853690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.853542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g" event={"ID":"dadcc031-8353-4a3f-a32e-44735c0adb3a","Type":"ContainerDied","Data":"25baa5245482cd432f694226afd8dfc45033fe3ade21cbc6e3d35dd2996f92c7"} Jul 14 20:18:19.853690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.853573 2574 scope.go:117] "RemoveContainer" containerID="6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae" Jul 14 20:18:19.863153 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.863138 2574 scope.go:117] "RemoveContainer" containerID="8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3" Jul 14 20:18:19.871022 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.871006 2574 scope.go:117] "RemoveContainer" containerID="6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae" Jul 14 20:18:19.871300 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:18:19.871282 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae\": container with ID starting with 6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae not found: ID does not exist" containerID="6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae" Jul 14 20:18:19.871348 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.871309 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae"} err="failed to get container status \"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae\": rpc error: code = NotFound desc = could not find container \"6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae\": container with ID starting with 6a439cbe5adb5cb8b883d298f5021aba5531a303b41aa84747a64619cec756ae not found: ID does not exist" Jul 14 20:18:19.871348 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.871325 2574 scope.go:117] "RemoveContainer" containerID="8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3" Jul 14 20:18:19.871573 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:18:19.871555 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3\": container with ID starting with 8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3 not found: ID does not exist" containerID="8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3" Jul 14 20:18:19.871618 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.871580 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3"} err="failed to get container status \"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3\": rpc error: code = NotFound desc = could not find container \"8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3\": container with ID starting with 8d2a4ac931366bcb90fb9c7e38652bbd297fee0c289b4f17194dddf952f5dfb3 not found: ID does not exist" Jul 14 20:18:19.877737 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.877718 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:18:19.884156 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:19.884138 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f02785bb-openshift-default-747f9d85bc-6hr7g"] Jul 14 20:18:20.600543 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:18:20.600482 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" path="/var/lib/kubelet/pods/dadcc031-8353-4a3f-a32e-44735c0adb3a/volumes" Jul 14 20:19:37.756140 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:37.756107 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:19:45.467805 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:45.467766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:19:51.788685 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.788652 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.788973 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.788983 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.788995 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789000 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789007 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.789047 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789013 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.789242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789073 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:19:51.789242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789082 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:19:51.789242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789089 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.789242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.789095 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dadcc031-8353-4a3f-a32e-44735c0adb3a" containerName="istio-proxy" Jul 14 20:19:51.792121 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.792103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.794781 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.794757 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-407eaab7-openshift-default-dockercfg-9qw9p\"" Jul 14 20:19:51.807701 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.807677 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:19:51.915026 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.914984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915026 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqjhq\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915262 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915262 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915262 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915262 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915409 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915409 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:51.915409 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:51.915337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.016613 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.016613 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqjhq\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.016852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.016852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.016852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017014 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017014 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017014 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.016978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017163 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017163 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017418 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017622 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.017787 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.018021 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.017839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.019327 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.019307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.019460 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.019443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.025009 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.024991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.025163 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.025143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqjhq\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq\") pod \"e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.102557 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.102464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:52.248025 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.247991 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:19:52.249222 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:19:52.249192 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5f5864_d508_4342_ae15_faacea5dcfee.slice/crio-6dbd8944f7c145e1e6b291abea430d95bdfffd961736d4f52cd7a6109fada750 WatchSource:0}: Error finding container 6dbd8944f7c145e1e6b291abea430d95bdfffd961736d4f52cd7a6109fada750: Status 404 returned error can't find the container with id 6dbd8944f7c145e1e6b291abea430d95bdfffd961736d4f52cd7a6109fada750 Jul 14 20:19:52.251395 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.251365 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:19:52.251459 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.251428 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:19:52.251543 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.251457 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:19:52.338648 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.338609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerStarted","Data":"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1"} Jul 14 20:19:52.338801 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.338668 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerStarted","Data":"6dbd8944f7c145e1e6b291abea430d95bdfffd961736d4f52cd7a6109fada750"} Jul 14 20:19:52.361914 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:52.361807 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podStartSLOduration=1.361787349 podStartE2EDuration="1.361787349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:19:52.358820474 +0000 UTC m=+1942.203157381" watchObservedRunningTime="2026-07-14 20:19:52.361787349 +0000 UTC m=+1942.206124224" Jul 14 20:19:53.102944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:53.102914 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:53.107712 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:53.107684 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:53.344096 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:53.344068 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:53.345062 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:53.345045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:19:54.056736 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:19:54.056694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:20:05.159110 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:05.159012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:20:08.345425 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.345384 2574 patch_prober.go:28] interesting pod/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 14 20:20:08.345902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.345445 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 14 20:20:08.422739 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.422710 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerID="2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1" exitCode=0 Jul 14 20:20:08.422876 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.422791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerDied","Data":"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1"} Jul 14 20:20:08.423198 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.423181 2574 scope.go:117] "RemoveContainer" containerID="2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1" Jul 14 20:20:08.424168 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.424138 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:08.424240 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.424223 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:08.424280 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:08.424272 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:09.432552 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:09.432516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerStarted","Data":"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329"} Jul 14 20:20:10.103663 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:10.103621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:10.103897 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:10.103870 2574 patch_prober.go:28] interesting pod/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 14 20:20:10.103978 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:10.103943 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 14 20:20:11.112929 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:11.112892 2574 patch_prober.go:28] interesting pod/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 14 20:20:11.113410 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:11.112951 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 14 20:20:12.103857 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:12.103813 2574 patch_prober.go:28] interesting pod/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 14 20:20:12.104054 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:12.103894 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 14 20:20:13.103766 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:13.103727 2574 patch_prober.go:28] interesting pod/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 14 20:20:13.104341 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:13.104308 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 14 20:20:14.106859 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:14.106829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:14.107230 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:14.107067 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:14.107901 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:14.107884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:20.662360 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:20.662329 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:20:32.938922 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:32.938844 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:20:32.939305 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:32.939048 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" containerID="cri-o://31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329" gracePeriod=30 Jul 14 20:20:38.197359 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.197334 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:38.311212 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311212 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311212 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311206 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqjhq\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311522 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311240 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311522 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311274 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311522 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311522 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311470 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311553 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311599 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311637 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token\") pod \"4c5f5864-d508-4342-ae15-faacea5dcfee\" (UID: \"4c5f5864-d508-4342-ae15-faacea5dcfee\") " Jul 14 20:20:38.311817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:20:38.312303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.311986 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.312303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.312006 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.312303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.312020 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c5f5864-d508-4342-ae15-faacea5dcfee-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.312303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.312033 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.312579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.312553 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:38.313613 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.313591 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:38.313796 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.313768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:20:38.313893 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.313840 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:20:38.313893 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.313839 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq" (OuterVolumeSpecName: "kube-api-access-cqjhq") pod "4c5f5864-d508-4342-ae15-faacea5dcfee" (UID: "4c5f5864-d508-4342-ae15-faacea5dcfee"). InnerVolumeSpecName "kube-api-access-cqjhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:20:38.412604 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.412579 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqjhq\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-kube-api-access-cqjhq\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.412604 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.412600 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.412756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.412610 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.412756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.412618 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.412756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.412626 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c5f5864-d508-4342-ae15-faacea5dcfee-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:20:38.583849 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.583781 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerID="31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329" exitCode=0 Jul 14 20:20:38.583849 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.583836 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" Jul 14 20:20:38.584007 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.583866 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerDied","Data":"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329"} Jul 14 20:20:38.584007 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.583928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7" event={"ID":"4c5f5864-d508-4342-ae15-faacea5dcfee","Type":"ContainerDied","Data":"6dbd8944f7c145e1e6b291abea430d95bdfffd961736d4f52cd7a6109fada750"} Jul 14 20:20:38.584007 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.583961 2574 scope.go:117] "RemoveContainer" containerID="31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329" Jul 14 20:20:38.593291 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.593277 2574 scope.go:117] "RemoveContainer" containerID="2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1" Jul 14 20:20:38.602308 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.602290 2574 scope.go:117] "RemoveContainer" containerID="31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329" Jul 14 20:20:38.602602 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:20:38.602581 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329\": container with ID starting with 31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329 not found: ID does not exist" containerID="31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329" Jul 14 20:20:38.602646 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.602614 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329"} err="failed to get container status \"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329\": rpc error: code = NotFound desc = could not find container \"31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329\": container with ID starting with 31502568649d923504e4b71b03707dc8653f17dc54c9a63f0d252fcb07c83329 not found: ID does not exist" Jul 14 20:20:38.602646 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.602631 2574 scope.go:117] "RemoveContainer" containerID="2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1" Jul 14 20:20:38.602904 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:20:38.602887 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1\": container with ID starting with 2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1 not found: ID does not exist" containerID="2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1" Jul 14 20:20:38.602943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.602911 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1"} err="failed to get container status \"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1\": rpc error: code = NotFound desc = could not find container \"2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1\": container with ID starting with 2000c638d259ea45d83eaf4c20324861eff57afd0f250fda9b922bce8838b5a1 not found: ID does not exist" Jul 14 20:20:38.616683 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.609804 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:20:38.619172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:38.619151 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-407eaab7-openshift-default-6777bc756c-wdtq7"] Jul 14 20:20:40.599920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:40.599885 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" path="/var/lib/kubelet/pods/4c5f5864-d508-4342-ae15-faacea5dcfee/volumes" Jul 14 20:20:53.038696 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.038656 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:20:53.039567 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.039541 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.039749 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.039735 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.039879 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.039868 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.039956 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.039947 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.040049 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.040039 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:20:53.040117 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.040109 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3554140c-3e66-4412-b0c0-cff892363c6c" containerName="istio-proxy" Jul 14 20:20:53.040450 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.040437 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.040621 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.040609 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5f5864-d508-4342-ae15-faacea5dcfee" containerName="istio-proxy" Jul 14 20:20:53.047811 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.047788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.051006 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.050979 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cbbc800e-openshift-default-dockercfg-xbl6v\"" Jul 14 20:20:53.054288 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.054265 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:20:53.136610 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.136803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.136803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.136803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.136803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.137005 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.137005 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k82m\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.137005 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.137005 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.136987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238447 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k82m\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.238672 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.239096 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.238996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.239096 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.239015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.239296 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.239273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.239372 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.239280 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.239372 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.239304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.241541 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.241520 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.241916 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.241894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.248073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.248052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.248597 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.248577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k82m\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m\") pod \"e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.360979 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.360897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:53.539174 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.539145 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:20:53.541201 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:20:53.541173 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473cad71_0a48_48d1_80b8_699fd026c8e2.slice/crio-edaeac295db201a023748e54fc9264b02aee68279ff388fb110548bf86ea30e1 WatchSource:0}: Error finding container edaeac295db201a023748e54fc9264b02aee68279ff388fb110548bf86ea30e1: Status 404 returned error can't find the container with id edaeac295db201a023748e54fc9264b02aee68279ff388fb110548bf86ea30e1 Jul 14 20:20:53.543821 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.543777 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:53.543913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.543844 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:53.543913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.543878 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:20:53.666627 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.666536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerStarted","Data":"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737"} Jul 14 20:20:53.666627 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.666606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerStarted","Data":"edaeac295db201a023748e54fc9264b02aee68279ff388fb110548bf86ea30e1"} Jul 14 20:20:53.688984 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:53.688928 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podStartSLOduration=0.688906284 podStartE2EDuration="688.906284ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:20:53.687832479 +0000 UTC m=+2003.532169398" watchObservedRunningTime="2026-07-14 20:20:53.688906284 +0000 UTC m=+2003.533243158" Jul 14 20:20:54.361093 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:54.361061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:54.362546 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:54.362520 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 20:20:54.362682 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:54.362566 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 20:20:55.365806 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:55.365773 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:55.676261 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:55.676177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:20:55.677249 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:20:55.677222 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:21:00.564558 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:00.564524 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:21:08.764272 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.764235 2574 generic.go:356] "Generic (PLEG): container finished" podID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerID="8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737" exitCode=0 Jul 14 20:21:08.764703 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.764282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerDied","Data":"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737"} Jul 14 20:21:08.764746 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.764708 2574 scope.go:117] "RemoveContainer" containerID="8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737" Jul 14 20:21:08.765720 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.765693 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:21:08.765772 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.765762 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:21:08.765813 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:08.765801 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:21:09.771292 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:09.771249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerStarted","Data":"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860"} Jul 14 20:21:10.361616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:10.361573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:21:10.362695 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:10.362668 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 20:21:10.362828 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:10.362718 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 20:21:10.564382 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:10.564343 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:21:10.676207 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:10.676117 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:21:11.361636 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:11.361595 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 20:21:11.362147 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:11.361658 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 20:21:12.361880 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:12.361840 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 20:21:12.362290 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:12.361912 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 20:21:13.361926 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:13.361888 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 14 20:21:13.362401 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:13.361965 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 14 20:21:14.365950 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:14.365919 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:21:14.366831 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:14.366810 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:21:25.063906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:25.063870 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:21:42.154383 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:21:42.154352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:22:30.771540 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:30.771512 2574 scope.go:117] "RemoveContainer" containerID="a4d3ccbd19387541df32bd9358a9acaa964336e8e693336ff65579e423d262ce" Jul 14 20:22:55.760845 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:55.760811 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:22:55.761422 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:55.761047 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" containerID="cri-o://6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860" gracePeriod=30 Jul 14 20:22:59.368523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:59.368469 2574 patch_prober.go:28] interesting pod/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:22:59.368883 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:59.368555 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:22:59.664959 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:22:59.664881 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:23:01.027069 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.026982 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:23:01.149540 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149510 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149604 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k82m\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149637 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149654 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149989 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149989 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149989 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149802 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:23:01.149989 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149839 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert\") pod \"473cad71-0a48-48d1-80b8-699fd026c8e2\" (UID: \"473cad71-0a48-48d1-80b8-699fd026c8e2\") " Jul 14 20:23:01.149989 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.149959 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:23:01.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150048 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:23:01.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150162 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150180 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150223 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.150441 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150346 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:23:01.150600 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.150527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data" (OuterVolumeSpecName: "istio-data") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:23:01.151985 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.151951 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m" (OuterVolumeSpecName: "kube-api-access-5k82m") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "kube-api-access-5k82m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:23:01.152164 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.152148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:23:01.152239 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.152221 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token" (OuterVolumeSpecName: "istio-token") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:23:01.152295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.152255 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "473cad71-0a48-48d1-80b8-699fd026c8e2" (UID: "473cad71-0a48-48d1-80b8-699fd026c8e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:23:01.250783 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250738 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.250783 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250774 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.250783 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250783 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.250783 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250791 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/473cad71-0a48-48d1-80b8-699fd026c8e2-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.250783 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250798 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.251079 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.250806 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k82m\" (UniqueName: \"kubernetes.io/projected/473cad71-0a48-48d1-80b8-699fd026c8e2-kube-api-access-5k82m\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:23:01.355566 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.355465 2574 generic.go:356] "Generic (PLEG): container finished" podID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerID="6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860" exitCode=0 Jul 14 20:23:01.355566 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.355544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerDied","Data":"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860"} Jul 14 20:23:01.355566 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.355549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" Jul 14 20:23:01.355838 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.355591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8" event={"ID":"473cad71-0a48-48d1-80b8-699fd026c8e2","Type":"ContainerDied","Data":"edaeac295db201a023748e54fc9264b02aee68279ff388fb110548bf86ea30e1"} Jul 14 20:23:01.355838 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.355615 2574 scope.go:117] "RemoveContainer" containerID="6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860" Jul 14 20:23:01.366724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.366702 2574 scope.go:117] "RemoveContainer" containerID="8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737" Jul 14 20:23:01.378942 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.378920 2574 scope.go:117] "RemoveContainer" containerID="6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860" Jul 14 20:23:01.379236 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:23:01.379216 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860\": container with ID starting with 6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860 not found: ID does not exist" containerID="6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860" Jul 14 20:23:01.379334 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.379250 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860"} err="failed to get container status \"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860\": rpc error: code = NotFound desc = could not find container \"6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860\": container with ID starting with 6e661ef544d753c4569e719d971a1d8536f6fb8454ca0ca8d938fd55c83e3860 not found: ID does not exist" Jul 14 20:23:01.379334 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.379276 2574 scope.go:117] "RemoveContainer" containerID="8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737" Jul 14 20:23:01.379533 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.379514 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:23:01.379685 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:23:01.379655 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737\": container with ID starting with 8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737 not found: ID does not exist" containerID="8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737" Jul 14 20:23:01.379811 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.379683 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737"} err="failed to get container status \"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737\": rpc error: code = NotFound desc = could not find container \"8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737\": container with ID starting with 8324db6088383f9af0c5d2a5f8da828037b64970d32b59e2088d504225b69737 not found: ID does not exist" Jul 14 20:23:01.382793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:01.382771 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cbbc800e-openshift-default-66bfd97b67-pzmk8"] Jul 14 20:23:02.599397 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:02.599368 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" path="/var/lib/kubelet/pods/473cad71-0a48-48d1-80b8-699fd026c8e2/volumes" Jul 14 20:23:13.159970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:13.159932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:23:50.713901 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.713872 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:23:50.714274 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714204 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.714274 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714215 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.714274 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714230 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.714274 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714235 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.714405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714296 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.714405 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.714306 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="473cad71-0a48-48d1-80b8-699fd026c8e2" containerName="istio-proxy" Jul 14 20:23:50.717296 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.717278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.719804 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.719782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d5be0-openshift-default-dockercfg-p9vpf\"" Jul 14 20:23:50.733048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.733028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:23:50.852721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.852888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.852888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.852888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.852888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.853031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9m2\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.853031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.852977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.853031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.853020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.853180 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.853096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.953893 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.953865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.953902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.953933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.953953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.953981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954056 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9m2\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954576 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954576 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954576 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954755 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.954755 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.954722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.956307 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.956286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.956678 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.956657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.961879 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.961856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:50.962137 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:50.962117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9m2\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2\") pod \"e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:51.027766 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.027703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:51.171516 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.171466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:23:51.173441 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:23:51.173414 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b99e559_e310_4a37_9b46_968c96dfa263.slice/crio-48c4d1cabc5c16eb671ddcd077dd90f8e276272b5444fc30dcd3f8681fa62c62 WatchSource:0}: Error finding container 48c4d1cabc5c16eb671ddcd077dd90f8e276272b5444fc30dcd3f8681fa62c62: Status 404 returned error can't find the container with id 48c4d1cabc5c16eb671ddcd077dd90f8e276272b5444fc30dcd3f8681fa62c62 Jul 14 20:23:51.175416 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.175376 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:51.175539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.175448 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:51.175539 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.175484 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:51.616147 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.616108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerStarted","Data":"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5"} Jul 14 20:23:51.616323 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.616167 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerStarted","Data":"48c4d1cabc5c16eb671ddcd077dd90f8e276272b5444fc30dcd3f8681fa62c62"} Jul 14 20:23:51.636620 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:51.636575 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podStartSLOduration=1.636559946 podStartE2EDuration="1.636559946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:23:51.634845055 +0000 UTC m=+2181.479181929" watchObservedRunningTime="2026-07-14 20:23:51.636559946 +0000 UTC m=+2181.480896821" Jul 14 20:23:52.028109 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:52.028078 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:52.029658 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:52.029637 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:23:52.029738 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:52.029691 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:23:53.032461 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:53.032419 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:53.626275 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:53.626240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:53.627217 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:53.627199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:23:58.790508 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.790464 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:23:58.794256 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.794234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.796824 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.796802 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eb3837-openshift-default-dockercfg-tns2q\"" Jul 14 20:23:58.805349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.805310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:23:58.923614 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923614 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923848 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923848 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp566\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923848 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923848 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923982 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923982 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:58.923982 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:58.923940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025222 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025222 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp566\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025523 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.025902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.026061 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.025993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.026578 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.026555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.027802 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.027776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.028007 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.027986 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.033886 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.033863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.040221 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.033887 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp566\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566\") pod \"e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.106224 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.106152 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:23:59.249853 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.249826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:23:59.252537 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:23:59.252506 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ba532a_be89_439e_9a86_35a837fc463c.slice/crio-1b13a45dabb241e9228d5253abb9c0cd73b37e8f5550b369e26e28e979b8c2bc WatchSource:0}: Error finding container 1b13a45dabb241e9228d5253abb9c0cd73b37e8f5550b369e26e28e979b8c2bc: Status 404 returned error can't find the container with id 1b13a45dabb241e9228d5253abb9c0cd73b37e8f5550b369e26e28e979b8c2bc Jul 14 20:23:59.255111 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.255078 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:59.255187 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.255145 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:59.255187 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.255175 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:23:59.656972 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.656935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerStarted","Data":"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108"} Jul 14 20:23:59.657272 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.657000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerStarted","Data":"1b13a45dabb241e9228d5253abb9c0cd73b37e8f5550b369e26e28e979b8c2bc"} Jul 14 20:23:59.678386 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:23:59.678327 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podStartSLOduration=1.6783070599999999 podStartE2EDuration="1.67830706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:23:59.677326876 +0000 UTC m=+2189.521663780" watchObservedRunningTime="2026-07-14 20:23:59.67830706 +0000 UTC m=+2189.522643929" Jul 14 20:24:00.106451 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:00.106420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:00.111756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:00.111733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:00.661599 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:00.661564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:00.662500 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:00.662469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:01.355255 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:01.355222 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:24:09.406040 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.404541 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:50700->10.134.0.48:15021: read: connection reset by peer" start-of-body= Jul 14 20:24:09.406040 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.404600 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:50700->10.134.0.48:15021: read: connection reset by peer" Jul 14 20:24:09.711438 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.711407 2574 generic.go:356] "Generic (PLEG): container finished" podID="5b99e559-e310-4a37-9b46-968c96dfa263" containerID="7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5" exitCode=0 Jul 14 20:24:09.711616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.711502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerDied","Data":"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5"} Jul 14 20:24:09.711945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.711931 2574 scope.go:117] "RemoveContainer" containerID="7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5" Jul 14 20:24:09.712945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.712914 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:09.713077 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.712996 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:09.713077 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:09.713044 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:10.723228 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:10.723172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerStarted","Data":"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966"} Jul 14 20:24:11.028695 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:11.028568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:24:11.029429 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:11.029392 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:24:11.029704 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:11.029648 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:24:12.028894 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:12.028855 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:24:12.029528 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:12.029467 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:24:13.029220 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:13.029178 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:24:13.029839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:13.029806 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:24:14.028774 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:14.028306 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:24:14.028774 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:14.028367 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:24:15.028901 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:15.028868 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:24:15.029598 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:15.029552 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 14 20:24:15.659215 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:15.659175 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:24:16.032299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:16.032265 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:24:16.032764 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:16.032712 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:24:16.033401 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:16.033372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:24:16.702665 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:16.702623 2574 patch_prober.go:28] interesting pod/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 14 20:24:16.702845 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:16.702691 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 14 20:24:17.772356 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.772280 2574 generic.go:356] "Generic (PLEG): container finished" podID="c9ba532a-be89-439e-9a86-35a837fc463c" containerID="318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108" exitCode=0 Jul 14 20:24:17.772765 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.772357 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerDied","Data":"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108"} Jul 14 20:24:17.772851 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.772836 2574 scope.go:117] "RemoveContainer" containerID="318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108" Jul 14 20:24:17.773827 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.773802 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:17.773902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.773865 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:17.773902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:17.773896 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:24:18.650645 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.650605 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:24:18.654726 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.654700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.657999 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.657976 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d5be0\"/\"openshift-service-ca.crt\"" Jul 14 20:24:18.657999 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.657991 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d5be0\"/\"kube-root-ca.crt\"" Jul 14 20:24:18.658145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.657990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d5be0\"/\"default-dockercfg-bmvx9\"" Jul 14 20:24:18.658145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.657976 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d5be0\"/\"test-model-8d5be0-kserve-self-signed-certs\"" Jul 14 20:24:18.662089 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.662062 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.721976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.722030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.722081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.722123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbjxn\" (UniqueName: \"kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.722164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.722515 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.722222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.779270 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.779235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerStarted","Data":"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33"} Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.822939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbjxn\" (UniqueName: \"kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.824189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.823851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.825420 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.825377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.826043 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.825649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.835846 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.834719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.835846 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.835374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.836698 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.836675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbjxn\" (UniqueName: \"kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn\") pod \"test-model-8d5be0-kserve-696c6b6c65-9dmhd\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:18.964673 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:18.964637 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:19.106777 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.106659 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:19.109584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.109157 2574 patch_prober.go:28] interesting pod/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 14 20:24:19.109584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.109223 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 14 20:24:19.158294 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.158269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:24:19.162371 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.162351 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:24:19.789046 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:19.788980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" event={"ID":"422e70ce-0fd5-4c8c-a332-91d85b3cf848","Type":"ContainerStarted","Data":"3e4198758339f4d8522633a4d6fc96470d3ec8d31e24ae75d8e40131fcf95011"} Jul 14 20:24:20.115823 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:20.115284 2574 patch_prober.go:28] interesting pod/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 14 20:24:20.115823 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:20.115356 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 14 20:24:20.796958 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:20.796918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" event={"ID":"422e70ce-0fd5-4c8c-a332-91d85b3cf848","Type":"ContainerStarted","Data":"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7"} Jul 14 20:24:20.817608 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:20.817552 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" podStartSLOduration=1.351359292 podStartE2EDuration="2.817530971s" totalImagesPullingTime="1.466171679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:24:20.814882101 +0000 UTC m=+2210.659218975" watchObservedRunningTime="2026-07-14 20:24:20.817530971 +0000 UTC m=+2210.661867846" Jul 14 20:24:21.106952 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:21.106906 2574 patch_prober.go:28] interesting pod/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 14 20:24:21.107151 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:21.106973 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 14 20:24:22.107121 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:22.107065 2574 patch_prober.go:28] interesting pod/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 14 20:24:22.107622 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:22.107142 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 14 20:24:23.110141 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:23.110108 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:23.110596 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:23.110405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:23.111083 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:23.111062 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:24:25.278852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:25.278811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:24:28.965449 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:28.965415 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:28.965920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:28.965568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:28.977972 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:28.977942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:24:29.861781 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:24:29.861753 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:25:26.252141 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.252104 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:25:26.255655 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.255638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.259140 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.259120 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb3837\"/\"kube-root-ca.crt\"" Jul 14 20:25:26.259258 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.259120 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb3837\"/\"default-dockercfg-7xn8b\"" Jul 14 20:25:26.259258 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.259118 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eb3837\"/\"test-model-eb3837-kserve-self-signed-certs\"" Jul 14 20:25:26.259258 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.259118 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eb3837\"/\"openshift-service-ca.crt\"" Jul 14 20:25:26.263909 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.263891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:25:26.445351 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.445546 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.445546 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.445546 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.445546 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7th\" (UniqueName: \"kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.445733 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.445596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.546867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.546867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.546980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7th\" (UniqueName: \"kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547373 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.547346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547458 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.547407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.547458 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.547406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.549479 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.549456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.550068 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.550045 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.554936 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.554913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7th\" (UniqueName: \"kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th\") pod \"test-model-eb3837-kserve-cbbbb4d45-7kdzv\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.565817 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.565798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:26.699842 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:26.699816 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:25:26.701208 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:25:26.701181 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod669ea9d8_092c_4dcc_9258_3dde92ce06c4.slice/crio-1af41af1c314ebe8e11d01365a3ef50fec7c10f089b3f01fa5fe24d0be78c3fd WatchSource:0}: Error finding container 1af41af1c314ebe8e11d01365a3ef50fec7c10f089b3f01fa5fe24d0be78c3fd: Status 404 returned error can't find the container with id 1af41af1c314ebe8e11d01365a3ef50fec7c10f089b3f01fa5fe24d0be78c3fd Jul 14 20:25:27.146022 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:27.145980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" event={"ID":"669ea9d8-092c-4dcc-9258-3dde92ce06c4","Type":"ContainerStarted","Data":"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30"} Jul 14 20:25:27.146196 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:27.146032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" event={"ID":"669ea9d8-092c-4dcc-9258-3dde92ce06c4","Type":"ContainerStarted","Data":"1af41af1c314ebe8e11d01365a3ef50fec7c10f089b3f01fa5fe24d0be78c3fd"} Jul 14 20:25:27.165832 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:27.165782 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" podStartSLOduration=1.165767631 podStartE2EDuration="1.165767631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:25:27.163437973 +0000 UTC m=+2277.007774848" watchObservedRunningTime="2026-07-14 20:25:27.165767631 +0000 UTC m=+2277.010104534" Jul 14 20:25:30.359804 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:30.359768 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:25:36.566847 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:36.566808 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:36.567358 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:36.566861 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:36.579311 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:36.579288 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:37.210660 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:37.210628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:25:44.251227 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:44.251192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:25:46.053141 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:46.053107 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:25:46.974568 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:46.974533 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:25:56.155898 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:25:56.155862 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:26:01.075448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.075403 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" podUID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" containerName="main" containerID="cri-o://6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7" gracePeriod=45 Jul 14 20:26:01.329097 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.329037 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:26:01.330863 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.330836 2574 generic.go:356] "Generic (PLEG): container finished" podID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" containerID="6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7" exitCode=0 Jul 14 20:26:01.330980 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.330875 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" Jul 14 20:26:01.330980 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.330919 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" event={"ID":"422e70ce-0fd5-4c8c-a332-91d85b3cf848","Type":"ContainerDied","Data":"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7"} Jul 14 20:26:01.330980 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.330964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd" event={"ID":"422e70ce-0fd5-4c8c-a332-91d85b3cf848","Type":"ContainerDied","Data":"3e4198758339f4d8522633a4d6fc96470d3ec8d31e24ae75d8e40131fcf95011"} Jul 14 20:26:01.331083 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.330982 2574 scope.go:117] "RemoveContainer" containerID="6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7" Jul 14 20:26:01.339626 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.339607 2574 scope.go:117] "RemoveContainer" containerID="6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7" Jul 14 20:26:01.339875 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:01.339858 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7\": container with ID starting with 6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7 not found: ID does not exist" containerID="6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7" Jul 14 20:26:01.339923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.339883 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7"} err="failed to get container status \"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7\": rpc error: code = NotFound desc = could not find container \"6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7\": container with ID starting with 6eccdd469fe18045443945588834d1c4d636bf41789a16d5e1aed948dbb42fa7 not found: ID does not exist" Jul 14 20:26:01.435667 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.435667 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbjxn\" (UniqueName: \"kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.435884 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435720 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.435884 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435739 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.435884 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.435884 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs\") pod \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\" (UID: \"422e70ce-0fd5-4c8c-a332-91d85b3cf848\") " Jul 14 20:26:01.436088 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.435889 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:01.436088 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.436001 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache" (OuterVolumeSpecName: "model-cache") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:01.436206 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.436079 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home" (OuterVolumeSpecName: "home") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:01.436206 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.436190 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tmp-dir\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.436304 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.436208 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-model-cache\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.436304 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.436221 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-home\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.438068 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.438042 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn" (OuterVolumeSpecName: "kube-api-access-rbjxn") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "kube-api-access-rbjxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:01.438068 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.438060 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm" (OuterVolumeSpecName: "dshm") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:01.438208 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.438104 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "422e70ce-0fd5-4c8c-a332-91d85b3cf848" (UID: "422e70ce-0fd5-4c8c-a332-91d85b3cf848"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:26:01.536910 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.536871 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/422e70ce-0fd5-4c8c-a332-91d85b3cf848-dshm\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.536910 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.536899 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/422e70ce-0fd5-4c8c-a332-91d85b3cf848-tls-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.536910 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.536915 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbjxn\" (UniqueName: \"kubernetes.io/projected/422e70ce-0fd5-4c8c-a332-91d85b3cf848-kube-api-access-rbjxn\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:01.653680 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.653654 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:26:01.658031 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.657998 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8d5be0/test-model-8d5be0-kserve-696c6b6c65-9dmhd"] Jul 14 20:26:01.998197 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:01.998155 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" podUID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" containerName="main" containerID="cri-o://c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30" gracePeriod=45 Jul 14 20:26:02.251336 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.251272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:26:02.337520 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.337459 2574 generic.go:356] "Generic (PLEG): container finished" podID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" containerID="c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30" exitCode=0 Jul 14 20:26:02.337682 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.337527 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" Jul 14 20:26:02.337682 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.337526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" event={"ID":"669ea9d8-092c-4dcc-9258-3dde92ce06c4","Type":"ContainerDied","Data":"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30"} Jul 14 20:26:02.337682 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.337653 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv" event={"ID":"669ea9d8-092c-4dcc-9258-3dde92ce06c4","Type":"ContainerDied","Data":"1af41af1c314ebe8e11d01365a3ef50fec7c10f089b3f01fa5fe24d0be78c3fd"} Jul 14 20:26:02.337682 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.337673 2574 scope.go:117] "RemoveContainer" containerID="c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30" Jul 14 20:26:02.342923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.342878 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343036 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.342952 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343036 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343006 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343137 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343043 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg7th\" (UniqueName: \"kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343192 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343145 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343192 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343156 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.343296 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm\") pod \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\" (UID: \"669ea9d8-092c-4dcc-9258-3dde92ce06c4\") " Jul 14 20:26:02.343296 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343260 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.343452 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343432 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home" (OuterVolumeSpecName: "home") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.343559 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343516 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-model-cache\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.343559 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343536 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tmp-dir\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.343559 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.343553 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-home\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.345428 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.345401 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:26:02.345553 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.345434 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm" (OuterVolumeSpecName: "dshm") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:02.345673 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.345656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th" (OuterVolumeSpecName: "kube-api-access-cg7th") pod "669ea9d8-092c-4dcc-9258-3dde92ce06c4" (UID: "669ea9d8-092c-4dcc-9258-3dde92ce06c4"). InnerVolumeSpecName "kube-api-access-cg7th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:02.347299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.347284 2574 scope.go:117] "RemoveContainer" containerID="c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30" Jul 14 20:26:02.347601 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:02.347578 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30\": container with ID starting with c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30 not found: ID does not exist" containerID="c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30" Jul 14 20:26:02.347661 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.347610 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30"} err="failed to get container status \"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30\": rpc error: code = NotFound desc = could not find container \"c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30\": container with ID starting with c759ac92fc2609d44e2ce8aadd4f99ffe31680fc1668c93ca8738a6b9626cf30 not found: ID does not exist" Jul 14 20:26:02.444685 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.444661 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669ea9d8-092c-4dcc-9258-3dde92ce06c4-dshm\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.444685 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.444685 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669ea9d8-092c-4dcc-9258-3dde92ce06c4-tls-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.444813 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.444696 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7th\" (UniqueName: \"kubernetes.io/projected/669ea9d8-092c-4dcc-9258-3dde92ce06c4-kube-api-access-cg7th\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:02.600190 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.600118 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" path="/var/lib/kubelet/pods/422e70ce-0fd5-4c8c-a332-91d85b3cf848/volumes" Jul 14 20:26:02.654769 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.654740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:26:02.656792 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:02.656769 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eb3837/test-model-eb3837-kserve-cbbbb4d45-7kdzv"] Jul 14 20:26:04.600392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:04.600359 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" path="/var/lib/kubelet/pods/669ea9d8-092c-4dcc-9258-3dde92ce06c4/volumes" Jul 14 20:26:05.658202 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:05.658161 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:26:12.930397 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:12.930363 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:26:12.931026 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:12.930593 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" containerID="cri-o://d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966" gracePeriod=30 Jul 14 20:26:16.035384 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:16.035349 2574 patch_prober.go:28] interesting pod/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:26:16.035765 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:16.035404 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:26:18.186101 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.186077 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:26:18.294061 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294029 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294232 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294232 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294103 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s9m2\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294232 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294206 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294252 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294289 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294312 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294344 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket\") pod \"5b99e559-e310-4a37-9b46-968c96dfa263\" (UID: \"5b99e559-e310-4a37-9b46-968c96dfa263\") " Jul 14 20:26:18.294424 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294410 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:18.294770 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294748 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.294770 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:26:18.294984 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:18.294984 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.294973 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:18.295426 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.295386 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:18.296577 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.296552 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:18.296977 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.296954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:18.297076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.297059 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:26:18.297250 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.297229 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2" (OuterVolumeSpecName: "kube-api-access-5s9m2") pod "5b99e559-e310-4a37-9b46-968c96dfa263" (UID: "5b99e559-e310-4a37-9b46-968c96dfa263"). InnerVolumeSpecName "kube-api-access-5s9m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:18.395844 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395816 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.395844 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395840 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9m2\" (UniqueName: \"kubernetes.io/projected/5b99e559-e310-4a37-9b46-968c96dfa263-kube-api-access-5s9m2\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.395844 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395850 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.396039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395859 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.396039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395866 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b99e559-e310-4a37-9b46-968c96dfa263-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.396039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395874 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.396039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395882 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b99e559-e310-4a37-9b46-968c96dfa263-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.396039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.395890 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b99e559-e310-4a37-9b46-968c96dfa263-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:18.424086 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.424055 2574 generic.go:356] "Generic (PLEG): container finished" podID="5b99e559-e310-4a37-9b46-968c96dfa263" containerID="d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966" exitCode=0 Jul 14 20:26:18.424190 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.424106 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" Jul 14 20:26:18.424190 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.424141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerDied","Data":"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966"} Jul 14 20:26:18.424265 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.424196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh" event={"ID":"5b99e559-e310-4a37-9b46-968c96dfa263","Type":"ContainerDied","Data":"48c4d1cabc5c16eb671ddcd077dd90f8e276272b5444fc30dcd3f8681fa62c62"} Jul 14 20:26:18.424265 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.424218 2574 scope.go:117] "RemoveContainer" containerID="d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966" Jul 14 20:26:18.434027 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.434010 2574 scope.go:117] "RemoveContainer" containerID="7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5" Jul 14 20:26:18.442531 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.442515 2574 scope.go:117] "RemoveContainer" containerID="d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966" Jul 14 20:26:18.442772 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:18.442755 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966\": container with ID starting with d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966 not found: ID does not exist" containerID="d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966" Jul 14 20:26:18.442834 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.442782 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966"} err="failed to get container status \"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966\": rpc error: code = NotFound desc = could not find container \"d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966\": container with ID starting with d8a0e09b7b10b7231e35d9aaf80557913b1dbf5f7e38ed76c5a62e572f746966 not found: ID does not exist" Jul 14 20:26:18.442879 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.442832 2574 scope.go:117] "RemoveContainer" containerID="7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5" Jul 14 20:26:18.443171 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:18.443135 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5\": container with ID starting with 7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5 not found: ID does not exist" containerID="7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5" Jul 14 20:26:18.443302 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.443178 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5"} err="failed to get container status \"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5\": rpc error: code = NotFound desc = could not find container \"7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5\": container with ID starting with 7f055d6dc82925ce6e1f0f863ba681e2a4df3e67e7a0e19b20f3f2bc19d706a5 not found: ID does not exist" Jul 14 20:26:18.446548 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.446529 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:26:18.450186 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.450168 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d5be0-openshift-default-746c75859c-xq9zh"] Jul 14 20:26:18.599888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:18.599823 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" path="/var/lib/kubelet/pods/5b99e559-e310-4a37-9b46-968c96dfa263/volumes" Jul 14 20:26:26.852883 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:26.852801 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:26:39.116016 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:39.115980 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:26:39.116425 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:39.116196 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" containerID="cri-o://d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33" gracePeriod=30 Jul 14 20:26:40.070902 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.070871 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:26:40.071224 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:40.071224 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071222 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" containerName="main" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071241 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" containerName="main" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071263 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" containerName="main" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071273 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" containerName="main" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071285 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071293 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071387 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071400 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="669ea9d8-092c-4dcc-9258-3dde92ce06c4" containerName="main" Jul 14 20:26:40.071417 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.071411 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="422e70ce-0fd5-4c8c-a332-91d85b3cf848" containerName="main" Jul 14 20:26:40.074721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.074700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.076980 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.076962 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-409b4f-openshift-default-dockercfg-q4llh\"" Jul 14 20:26:40.086521 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.086471 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:26:40.169711 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.169654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.170319 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.170294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.170461 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.170445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxsj\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.170640 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.170621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.170818 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.170802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.170954 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.170938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.171077 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.171063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.171196 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.171183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.171313 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.171299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.272861 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.272817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.272894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.272925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.272950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxsj\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273520 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273520 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273467 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273680 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.273737 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.273721 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.275862 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.275838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.276034 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.276009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.282287 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.282262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.282420 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.282377 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxsj\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj\") pod \"e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.386172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.386085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:40.527052 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.527023 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:26:40.528655 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:26:40.528629 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09915653_8689_4ed4_89e5_b3001673e45b.slice/crio-1b73507b458370d188da5357a6788dbce6dc57247b9bd99030eb8b8311f86b4e WatchSource:0}: Error finding container 1b73507b458370d188da5357a6788dbce6dc57247b9bd99030eb8b8311f86b4e: Status 404 returned error can't find the container with id 1b73507b458370d188da5357a6788dbce6dc57247b9bd99030eb8b8311f86b4e Jul 14 20:26:40.530763 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.530732 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:40.530867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.530807 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:40.530867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.530854 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:40.538651 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:40.538618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerStarted","Data":"1b73507b458370d188da5357a6788dbce6dc57247b9bd99030eb8b8311f86b4e"} Jul 14 20:26:41.544803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:41.544769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerStarted","Data":"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6"} Jul 14 20:26:41.565720 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:41.565672 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podStartSLOduration=1.5656554919999999 podStartE2EDuration="1.565655492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:41.563074355 +0000 UTC m=+2351.407411255" watchObservedRunningTime="2026-07-14 20:26:41.565655492 +0000 UTC m=+2351.409992366" Jul 14 20:26:42.386507 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:42.386460 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:42.391561 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:42.391535 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:42.549566 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:42.549531 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:42.550614 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:42.550592 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:44.369057 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.369028 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:26:44.409244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409244 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409218 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409248 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp566\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409274 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409315 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409400 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token\") pod \"c9ba532a-be89-439e-9a86-35a837fc463c\" (UID: \"c9ba532a-be89-439e-9a86-35a837fc463c\") " Jul 14 20:26:44.409466 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409422 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:44.409961 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:26:44.409961 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409740 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ba532a-be89-439e-9a86-35a837fc463c-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.409961 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409761 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.409961 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409808 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:44.410242 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.409962 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:44.410438 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.410395 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:44.411891 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.411867 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:26:44.412054 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.412038 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566" (OuterVolumeSpecName: "kube-api-access-pp566") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "kube-api-access-pp566". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:44.412165 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.412150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:26:44.412255 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.412242 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9ba532a-be89-439e-9a86-35a837fc463c" (UID: "c9ba532a-be89-439e-9a86-35a837fc463c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:26:44.510849 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510818 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.510849 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510844 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.510849 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510855 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ba532a-be89-439e-9a86-35a837fc463c-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.511076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510864 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.511076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510872 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.511076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510880 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp566\" (UniqueName: \"kubernetes.io/projected/c9ba532a-be89-439e-9a86-35a837fc463c-kube-api-access-pp566\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.511076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.510889 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ba532a-be89-439e-9a86-35a837fc463c-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:26:44.561775 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.561743 2574 generic.go:356] "Generic (PLEG): container finished" podID="c9ba532a-be89-439e-9a86-35a837fc463c" containerID="d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33" exitCode=0 Jul 14 20:26:44.561920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.561798 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" Jul 14 20:26:44.561920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.561833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerDied","Data":"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33"} Jul 14 20:26:44.561920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.561891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc" event={"ID":"c9ba532a-be89-439e-9a86-35a837fc463c","Type":"ContainerDied","Data":"1b13a45dabb241e9228d5253abb9c0cd73b37e8f5550b369e26e28e979b8c2bc"} Jul 14 20:26:44.561920 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.561916 2574 scope.go:117] "RemoveContainer" containerID="d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33" Jul 14 20:26:44.571615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.571599 2574 scope.go:117] "RemoveContainer" containerID="318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108" Jul 14 20:26:44.580200 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.580182 2574 scope.go:117] "RemoveContainer" containerID="d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33" Jul 14 20:26:44.580437 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:44.580420 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33\": container with ID starting with d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33 not found: ID does not exist" containerID="d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33" Jul 14 20:26:44.580525 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.580448 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33"} err="failed to get container status \"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33\": rpc error: code = NotFound desc = could not find container \"d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33\": container with ID starting with d0ff2a3fdda57761c503e3fc38406db007b64e8f67e4799aac62b263ef025b33 not found: ID does not exist" Jul 14 20:26:44.580525 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.580463 2574 scope.go:117] "RemoveContainer" containerID="318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108" Jul 14 20:26:44.580719 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:26:44.580703 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108\": container with ID starting with 318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108 not found: ID does not exist" containerID="318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108" Jul 14 20:26:44.580760 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.580722 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108"} err="failed to get container status \"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108\": rpc error: code = NotFound desc = could not find container \"318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108\": container with ID starting with 318b8e00f15e1258753351fa314b26ff4fc5a6aaf0fc8d326e7a97bd22de2108 not found: ID does not exist" Jul 14 20:26:44.587219 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.587197 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:26:44.599985 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:44.599967 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3837-openshift-default-6999bc566c-b9xfc"] Jul 14 20:26:46.457119 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:46.457075 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:26:46.599377 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:46.599344 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" path="/var/lib/kubelet/pods/c9ba532a-be89-439e-9a86-35a837fc463c/volumes" Jul 14 20:26:48.903545 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.903513 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:26:48.912753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.912721 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:26:48.912948 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.912932 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:26:48.913078 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.913064 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:26:48.913184 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.913171 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:26:48.920520 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.913364 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:26:48.924944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.924921 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b99e559-e310-4a37-9b46-968c96dfa263" containerName="istio-proxy" Jul 14 20:26:48.931513 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.929898 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:26:48.931513 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.930013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:48.932910 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:48.932886 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0f00f6-openshift-default-dockercfg-mqrxq\"" Jul 14 20:26:49.048207 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj4j6\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048380 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.048615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.048450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.149814 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.149778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.149997 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.149828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.149997 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.149963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150084 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.149999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj4j6\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150084 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150084 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150235 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150345 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150404 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150382 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150578 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.150996 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.150971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.151106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.151048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.151153 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.151106 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.152446 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.152422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.152668 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.152651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.162291 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.162236 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj4j6\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.163904 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.163888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token\") pod \"e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.242408 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.242376 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:49.428945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.428781 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:26:49.431652 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.431154 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:49.431652 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.431226 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:49.431652 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.431291 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:49.591793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.591754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerStarted","Data":"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216"} Jul 14 20:26:49.591983 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.591816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerStarted","Data":"0d787d9bbb9be3d2ea85687b0d7ec7ccbe615d63296bc917aaf18d6ae020d14c"} Jul 14 20:26:49.612957 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:49.612887 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podStartSLOduration=1.612864032 podStartE2EDuration="1.612864032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:49.608976182 +0000 UTC m=+2359.453313057" watchObservedRunningTime="2026-07-14 20:26:49.612864032 +0000 UTC m=+2359.457200910" Jul 14 20:26:50.243528 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:50.243469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:50.248170 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:50.248146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:50.600655 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:50.600525 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:50.600655 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:50.600607 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:26:54.624548 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.624428 2574 generic.go:356] "Generic (PLEG): container finished" podID="09915653-8689-4ed4-89e5-b3001673e45b" containerID="080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6" exitCode=0 Jul 14 20:26:54.624548 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.624517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerDied","Data":"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6"} Jul 14 20:26:54.625107 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.625028 2574 scope.go:117] "RemoveContainer" containerID="080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6" Jul 14 20:26:54.626066 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.626040 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:54.626128 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.626103 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:54.626164 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:54.626135 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:26:55.633391 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:55.633354 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerStarted","Data":"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393"} Jul 14 20:26:56.386388 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:56.386351 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:56.387456 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:56.387429 2574 patch_prober.go:28] interesting pod/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:26:56.387595 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:56.387481 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:26:57.386431 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:57.386397 2574 patch_prober.go:28] interesting pod/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:26:57.386840 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:57.386465 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:26:57.549604 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:57.549567 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:26:58.386624 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:58.386588 2574 patch_prober.go:28] interesting pod/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:26:58.387015 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:58.386643 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:26:59.386574 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:59.386545 2574 patch_prober.go:28] interesting pod/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 14 20:26:59.386770 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:26:59.386594 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 14 20:27:00.390666 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:00.390630 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:27:00.391418 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:00.391401 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:27:02.166318 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:02.165543 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:27:10.720281 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.720200 2574 generic.go:356] "Generic (PLEG): container finished" podID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerID="df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216" exitCode=0 Jul 14 20:27:10.720723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.720276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerDied","Data":"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216"} Jul 14 20:27:10.720723 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.720694 2574 scope.go:117] "RemoveContainer" containerID="df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216" Jul 14 20:27:10.721670 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.721642 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:10.721779 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.721707 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:10.721779 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:10.721742 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:11.732243 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:11.732205 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerStarted","Data":"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec"} Jul 14 20:27:12.242749 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:12.242711 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:27:12.243063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:12.243040 2574 patch_prober.go:28] interesting pod/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 14 20:27:12.243135 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:12.243092 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 14 20:27:13.242909 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:13.242876 2574 patch_prober.go:28] interesting pod/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 14 20:27:13.243272 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:13.242936 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 14 20:27:14.243022 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:14.242987 2574 patch_prober.go:28] interesting pod/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 14 20:27:14.243456 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:14.243038 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 14 20:27:15.243675 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:15.243638 2574 patch_prober.go:28] interesting pod/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 14 20:27:15.244145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:15.243703 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 14 20:27:16.247302 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:16.247271 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:27:16.247803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:16.247675 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:27:16.248236 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:16.248216 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:27:19.859095 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:19.859058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:27:30.152657 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:30.152620 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:27:30.153119 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:30.152841 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" containerID="cri-o://1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393" gracePeriod=30 Jul 14 20:27:30.393759 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:30.393726 2574 patch_prober.go:28] interesting pod/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:27:30.393925 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:30.393786 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:27:35.404228 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.404202 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:27:35.487758 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.487731 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.487943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.487790 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.487943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.487840 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.487943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.487868 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.487943 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.487902 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.488155 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488023 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.488155 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488064 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpxsj\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.488155 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488131 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.488316 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488165 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:27:35.488316 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488215 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket\") pod \"09915653-8689-4ed4-89e5-b3001673e45b\" (UID: \"09915653-8689-4ed4-89e5-b3001673e45b\") " Jul 14 20:27:35.488316 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:27:35.488479 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:27:35.488707 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488657 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09915653-8689-4ed4-89e5-b3001673e45b-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.488707 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data" (OuterVolumeSpecName: "istio-data") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:27:35.488707 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488683 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.488891 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488720 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.488891 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.488695 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:27:35.490366 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.490339 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:27:35.490529 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.490478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:27:35.490756 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.490731 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj" (OuterVolumeSpecName: "kube-api-access-hpxsj") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "kube-api-access-hpxsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:27:35.490825 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.490753 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token" (OuterVolumeSpecName: "istio-token") pod "09915653-8689-4ed4-89e5-b3001673e45b" (UID: "09915653-8689-4ed4-89e5-b3001673e45b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:27:35.589480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589451 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09915653-8689-4ed4-89e5-b3001673e45b-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.589480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589475 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.589480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589507 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.589717 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589517 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.589717 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589525 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09915653-8689-4ed4-89e5-b3001673e45b-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.589717 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.589533 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxsj\" (UniqueName: \"kubernetes.io/projected/09915653-8689-4ed4-89e5-b3001673e45b-kube-api-access-hpxsj\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:27:35.858183 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.858092 2574 generic.go:356] "Generic (PLEG): container finished" podID="09915653-8689-4ed4-89e5-b3001673e45b" containerID="1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393" exitCode=0 Jul 14 20:27:35.858183 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.858134 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" Jul 14 20:27:35.858354 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.858183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerDied","Data":"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393"} Jul 14 20:27:35.858354 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.858246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk" event={"ID":"09915653-8689-4ed4-89e5-b3001673e45b","Type":"ContainerDied","Data":"1b73507b458370d188da5357a6788dbce6dc57247b9bd99030eb8b8311f86b4e"} Jul 14 20:27:35.858354 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.858271 2574 scope.go:117] "RemoveContainer" containerID="1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393" Jul 14 20:27:35.867888 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.867870 2574 scope.go:117] "RemoveContainer" containerID="080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6" Jul 14 20:27:35.876461 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.876446 2574 scope.go:117] "RemoveContainer" containerID="1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393" Jul 14 20:27:35.876737 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:27:35.876719 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393\": container with ID starting with 1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393 not found: ID does not exist" containerID="1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393" Jul 14 20:27:35.876809 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.876746 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393"} err="failed to get container status \"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393\": rpc error: code = NotFound desc = could not find container \"1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393\": container with ID starting with 1589e3f69c8eacbd52332aa86215cd06d512237ab331e0a36238be88dff06393 not found: ID does not exist" Jul 14 20:27:35.876809 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.876769 2574 scope.go:117] "RemoveContainer" containerID="080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6" Jul 14 20:27:35.877452 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:27:35.877108 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6\": container with ID starting with 080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6 not found: ID does not exist" containerID="080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6" Jul 14 20:27:35.877452 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.877139 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6"} err="failed to get container status \"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6\": rpc error: code = NotFound desc = could not find container \"080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6\": container with ID starting with 080db7d22fa3716960bbdfb727da3e0ec146d4163881384dd7d3d7e69821e3c6 not found: ID does not exist" Jul 14 20:27:35.879847 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.879828 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:27:35.883780 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:35.883758 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-409b4f-openshift-default-74fb56b577-c2slk"] Jul 14 20:27:36.599185 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:36.599153 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09915653-8689-4ed4-89e5-b3001673e45b" path="/var/lib/kubelet/pods/09915653-8689-4ed4-89e5-b3001673e45b/volumes" Jul 14 20:27:44.158435 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:44.158401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:27:56.343577 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:56.343472 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:27:56.344076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:56.343736 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" containerID="cri-o://a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec" gracePeriod=30 Jul 14 20:27:57.300842 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.300804 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:27:57.301315 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301299 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.301376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301319 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.301376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301340 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.301376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301349 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.301470 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301441 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ba532a-be89-439e-9a86-35a837fc463c" containerName="istio-proxy" Jul 14 20:27:57.301470 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301454 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.301470 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.301466 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="09915653-8689-4ed4-89e5-b3001673e45b" containerName="istio-proxy" Jul 14 20:27:57.315249 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.315213 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:27:57.315412 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.315339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.317706 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.317682 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-10ef4d-openshift-default-dockercfg-kcxkt\"" Jul 14 20:27:57.475648 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.475648 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4rdg\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.476048 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.475920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4rdg\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577448 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.577812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.578012 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.577979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.578084 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.578032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.578217 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.578200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.579534 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.579512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.579770 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.579752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.586087 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.586059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.586173 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.586147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4rdg\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg\") pod \"e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.625324 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.625298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:57.760144 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.760115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:27:57.761684 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:27:57.761658 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39e3a4_90c3_4805_8fec_bfda27a48398.slice/crio-4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28 WatchSource:0}: Error finding container 4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28: Status 404 returned error can't find the container with id 4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28 Jul 14 20:27:57.764052 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.764021 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:57.764125 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.764082 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:57.764125 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.764108 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:27:57.982576 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.982533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerStarted","Data":"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a"} Jul 14 20:27:57.982743 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:57.982601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerStarted","Data":"4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28"} Jul 14 20:27:58.003967 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:58.003902 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podStartSLOduration=1.003881931 podStartE2EDuration="1.003881931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:27:57.999665119 +0000 UTC m=+2427.844002016" watchObservedRunningTime="2026-07-14 20:27:58.003881931 +0000 UTC m=+2427.848218807" Jul 14 20:27:58.626517 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:58.626449 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:58.627858 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:58.627830 2574 patch_prober.go:28] interesting pod/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:27:58.628002 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:58.627875 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:27:59.630072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:59.630043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:59.993353 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:59.993323 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:27:59.994290 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:27:59.994269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:28:01.250087 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.250045 2574 patch_prober.go:28] interesting pod/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:28:01.250432 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.250106 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:28:01.598298 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.598271 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:28:01.715793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715764 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715831 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715871 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715909 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj4j6\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715946 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.715970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.716271 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.715994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.716271 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.716028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy\") pod \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\" (UID: \"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4\") " Jul 14 20:28:01.716383 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.716319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:28:01.716383 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.716365 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:28:01.717036 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.716926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:28:01.717036 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.717009 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:28:01.717219 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.717087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:28:01.717219 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.717145 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.717219 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.717175 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.719425 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.719275 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:28:01.724873 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.724848 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:28:01.725426 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.725402 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6" (OuterVolumeSpecName: "kube-api-access-jj4j6") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "kube-api-access-jj4j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:28:01.725547 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.725425 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" (UID: "cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:28:01.818120 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818090 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818120 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818116 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818120 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818124 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818133 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818162 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj4j6\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-kube-api-access-jj4j6\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818172 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:01.818303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:01.818180 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:28:02.005166 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.005131 2574 generic.go:356] "Generic (PLEG): container finished" podID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerID="a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec" exitCode=0 Jul 14 20:28:02.005309 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.005207 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" Jul 14 20:28:02.005309 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.005222 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerDied","Data":"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec"} Jul 14 20:28:02.005404 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.005317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2" event={"ID":"cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4","Type":"ContainerDied","Data":"0d787d9bbb9be3d2ea85687b0d7ec7ccbe615d63296bc917aaf18d6ae020d14c"} Jul 14 20:28:02.005404 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.005349 2574 scope.go:117] "RemoveContainer" containerID="a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec" Jul 14 20:28:02.016250 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.016228 2574 scope.go:117] "RemoveContainer" containerID="df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216" Jul 14 20:28:02.025221 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.025208 2574 scope.go:117] "RemoveContainer" containerID="a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec" Jul 14 20:28:02.025509 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:28:02.025467 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec\": container with ID starting with a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec not found: ID does not exist" containerID="a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec" Jul 14 20:28:02.025569 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.025518 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec"} err="failed to get container status \"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec\": rpc error: code = NotFound desc = could not find container \"a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec\": container with ID starting with a516630c3722543451b71cc065a7c23297031ff7d37fecc3e3ae3d4a0700f7ec not found: ID does not exist" Jul 14 20:28:02.025569 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.025537 2574 scope.go:117] "RemoveContainer" containerID="df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216" Jul 14 20:28:02.025796 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:28:02.025764 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216\": container with ID starting with df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216 not found: ID does not exist" containerID="df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216" Jul 14 20:28:02.025855 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.025804 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216"} err="failed to get container status \"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216\": rpc error: code = NotFound desc = could not find container \"df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216\": container with ID starting with df1f02d1e3511e31e8631b7fb4aeb313e3f95472c01d5bc43bfdb1f56a566216 not found: ID does not exist" Jul 14 20:28:02.031374 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.031353 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:28:02.035167 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.035146 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0f00f6-openshift-default-8444d6b545-n5fx2"] Jul 14 20:28:02.599513 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:02.599461 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" path="/var/lib/kubelet/pods/cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4/volumes" Jul 14 20:28:04.458435 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:04.458398 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:28:05.134119 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134083 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:28:05.134601 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134579 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.134601 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134601 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.134778 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134620 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.134778 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134628 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.134778 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134721 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.134778 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.134733 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc86a18e-5dfe-47ce-a7f2-4d08a4a9a6b4" containerName="istio-proxy" Jul 14 20:28:05.139327 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.139304 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.142648 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.142625 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d04b66-openshift-default-dockercfg-9rzls\"" Jul 14 20:28:05.150070 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.150045 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:28:05.250806 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.250772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.250806 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.250812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.250847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.250919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.250971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.251014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.251034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.251051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.251441 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.251154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm9f\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352797 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352797 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352797 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm9f\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352797 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.352994 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.352939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.353082 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.353054 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.353216 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.353118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.353284 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.353239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.354807 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.354788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.355153 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.355136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.364812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.364784 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.364969 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.364948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm9f\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f\") pod \"e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.450811 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.450770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:05.591361 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.591335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:28:05.593298 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:28:05.593270 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d212c_2ac2_4a98_ac8a_1c05f7b4db69.slice/crio-09bb2a3022d24814ff20013858881c74376b1783e2069e6db7cb1e9dbc42c63f WatchSource:0}: Error finding container 09bb2a3022d24814ff20013858881c74376b1783e2069e6db7cb1e9dbc42c63f: Status 404 returned error can't find the container with id 09bb2a3022d24814ff20013858881c74376b1783e2069e6db7cb1e9dbc42c63f Jul 14 20:28:05.595282 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.595251 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:05.595378 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.595327 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:05.595444 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:05.595377 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:06.043121 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:06.043079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerStarted","Data":"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9"} Jul 14 20:28:06.043330 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:06.043175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerStarted","Data":"09bb2a3022d24814ff20013858881c74376b1783e2069e6db7cb1e9dbc42c63f"} Jul 14 20:28:06.064652 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:06.064605 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podStartSLOduration=1.064588786 podStartE2EDuration="1.064588786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:28:06.062222641 +0000 UTC m=+2435.906559515" watchObservedRunningTime="2026-07-14 20:28:06.064588786 +0000 UTC m=+2435.908925638" Jul 14 20:28:06.451129 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:06.451098 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:06.456588 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:06.456565 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:07.048581 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:07.048545 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:07.049471 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:07.049453 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:13.084033 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.083944 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerID="db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a" exitCode=0 Jul 14 20:28:13.084033 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.084019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerDied","Data":"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a"} Jul 14 20:28:13.084464 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.084410 2574 scope.go:117] "RemoveContainer" containerID="db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a" Jul 14 20:28:13.085605 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.085570 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:13.085725 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.085656 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:13.085725 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:13.085703 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:14.092098 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:14.092061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerStarted","Data":"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00"} Jul 14 20:28:14.627305 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:14.627266 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:28:14.627537 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:14.627325 2574 patch_prober.go:28] interesting pod/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:28:14.627537 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:14.627393 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:28:14.993862 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:14.993828 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:28:15.626759 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:15.626718 2574 patch_prober.go:28] interesting pod/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:28:15.627254 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:15.626783 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:28:16.628536 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:16.627919 2574 patch_prober.go:28] interesting pod/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:28:16.628536 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:16.627978 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:28:17.625734 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:17.625693 2574 patch_prober.go:28] interesting pod/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 14 20:28:17.625925 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:17.625780 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 14 20:28:18.630482 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:18.630451 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:28:18.631850 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:18.631824 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:28:20.135779 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.135748 2574 generic.go:356] "Generic (PLEG): container finished" podID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerID="f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9" exitCode=0 Jul 14 20:28:20.136106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.135820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerDied","Data":"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9"} Jul 14 20:28:20.136246 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.136232 2574 scope.go:117] "RemoveContainer" containerID="f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9" Jul 14 20:28:20.137210 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.137184 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:20.137270 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.137251 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:20.137303 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.137287 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 14 20:28:20.343970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.343867 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:28:20.347430 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.347404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.349906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.349883 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"default-dockercfg-6jznz\"" Jul 14 20:28:20.350038 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.349904 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"openshift-service-ca.crt\"" Jul 14 20:28:20.350334 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.350313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"auth-test-model-10ef4d-kserve-self-signed-certs\"" Jul 14 20:28:20.350440 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.350378 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-10ef4d\"/\"kube-root-ca.crt\"" Jul 14 20:28:20.356970 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.356939 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:28:20.385065 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqg4w\" (UniqueName: \"kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.385189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.385189 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.385283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.385283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.385283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.385231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486855 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486855 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486855 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486998 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqg4w\" (UniqueName: \"kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.486998 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.486982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.487091 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.487071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.487200 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.487185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.487289 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.487268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.489157 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.489134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.489552 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.489532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.495769 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.495748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqg4w\" (UniqueName: \"kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w\") pod \"auth-test-model-10ef4d-kserve-7644db87f5-fh5t7\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.659453 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.659366 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:20.792771 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.792744 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:28:20.794114 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:28:20.794087 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05feb1b9_2478_4af3_8726_74399493b081.slice/crio-bfde1e194139d46a9989dea4dc3499b7a7bf097c99438256b1f50a92f639ae9b WatchSource:0}: Error finding container bfde1e194139d46a9989dea4dc3499b7a7bf097c99438256b1f50a92f639ae9b: Status 404 returned error can't find the container with id bfde1e194139d46a9989dea4dc3499b7a7bf097c99438256b1f50a92f639ae9b Jul 14 20:28:20.953972 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:20.953933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:28:21.142628 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.142584 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" event={"ID":"05feb1b9-2478-4af3-8726-74399493b081","Type":"ContainerStarted","Data":"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012"} Jul 14 20:28:21.143069 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.142636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" event={"ID":"05feb1b9-2478-4af3-8726-74399493b081","Type":"ContainerStarted","Data":"bfde1e194139d46a9989dea4dc3499b7a7bf097c99438256b1f50a92f639ae9b"} Jul 14 20:28:21.144715 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.144689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerStarted","Data":"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2"} Jul 14 20:28:21.159022 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.158971 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" podStartSLOduration=1.158955468 podStartE2EDuration="1.158955468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:28:21.158004491 +0000 UTC m=+2451.002341378" watchObservedRunningTime="2026-07-14 20:28:21.158955468 +0000 UTC m=+2451.003292341" Jul 14 20:28:21.451329 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.451293 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:21.452558 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.452529 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 14 20:28:21.452690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:21.452576 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 14 20:28:22.049145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:22.049108 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:22.451941 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:22.451906 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 14 20:28:22.452299 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:22.451979 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 14 20:28:23.451861 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:23.451823 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 14 20:28:23.452039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:23.451879 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 14 20:28:24.451330 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:24.451296 2574 patch_prober.go:28] interesting pod/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 14 20:28:24.451542 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:24.451351 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 14 20:28:25.455728 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:25.455698 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:25.456695 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:25.456673 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:28:30.659742 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:30.659707 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:30.660172 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:30.659759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:30.672356 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:30.672320 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:31.217473 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:31.217438 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:28:31.753070 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:31.753040 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:28:43.750944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:43.750906 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:28:54.157657 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:28:54.157622 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:29:12.755839 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:29:12.755808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:29:28.264667 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:29:28.264585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:29:45.358947 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:29:45.358915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:30:01.466019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:01.465981 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:30:17.960670 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:17.960635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:30:29.478134 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:29.478103 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:30:30.767316 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:30.767284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:30:42.056331 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:42.056288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:30:44.499181 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.499130 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" podUID="05feb1b9-2478-4af3-8726-74399493b081" containerName="main" containerID="cri-o://3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012" gracePeriod=45 Jul 14 20:30:44.748030 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.748006 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:30:44.793399 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793399 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793393 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793426 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793457 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793466 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:44.793619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793548 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqg4w\" (UniqueName: \"kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home\") pod \"05feb1b9-2478-4af3-8726-74399493b081\" (UID: \"05feb1b9-2478-4af3-8726-74399493b081\") " Jul 14 20:30:44.793879 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793695 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache" (OuterVolumeSpecName: "model-cache") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:44.793945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.793928 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home" (OuterVolumeSpecName: "home") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:44.794021 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.794000 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-tmp-dir\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.794072 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.794027 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-model-cache\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.795926 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.795901 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w" (OuterVolumeSpecName: "kube-api-access-kqg4w") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "kube-api-access-kqg4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:30:44.796061 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.796024 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:30:44.796061 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.796034 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm" (OuterVolumeSpecName: "dshm") pod "05feb1b9-2478-4af3-8726-74399493b081" (UID: "05feb1b9-2478-4af3-8726-74399493b081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:44.895132 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.895098 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqg4w\" (UniqueName: \"kubernetes.io/projected/05feb1b9-2478-4af3-8726-74399493b081-kube-api-access-kqg4w\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.895132 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.895125 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-home\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.895132 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.895136 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05feb1b9-2478-4af3-8726-74399493b081-dshm\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.895349 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.895144 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05feb1b9-2478-4af3-8726-74399493b081-tls-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.904811 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.904777 2574 generic.go:356] "Generic (PLEG): container finished" podID="05feb1b9-2478-4af3-8726-74399493b081" containerID="3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012" exitCode=0 Jul 14 20:30:44.904963 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.904827 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" Jul 14 20:30:44.904963 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.904857 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" event={"ID":"05feb1b9-2478-4af3-8726-74399493b081","Type":"ContainerDied","Data":"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012"} Jul 14 20:30:44.904963 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.904903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7" event={"ID":"05feb1b9-2478-4af3-8726-74399493b081","Type":"ContainerDied","Data":"bfde1e194139d46a9989dea4dc3499b7a7bf097c99438256b1f50a92f639ae9b"} Jul 14 20:30:44.904963 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.904926 2574 scope.go:117] "RemoveContainer" containerID="3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012" Jul 14 20:30:44.918700 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.918666 2574 scope.go:117] "RemoveContainer" containerID="3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012" Jul 14 20:30:44.919219 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:30:44.919172 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012\": container with ID starting with 3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012 not found: ID does not exist" containerID="3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012" Jul 14 20:30:44.919316 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.919242 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012"} err="failed to get container status \"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012\": rpc error: code = NotFound desc = could not find container \"3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012\": container with ID starting with 3552a5092c24fcf0535559ad5e0a04683a2b0c5e431e3e8efc30273ef1d46012 not found: ID does not exist" Jul 14 20:30:44.934967 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.934939 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:30:44.942039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:44.942016 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-10ef4d/auth-test-model-10ef4d-kserve-7644db87f5-fh5t7"] Jul 14 20:30:46.599611 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:46.599578 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05feb1b9-2478-4af3-8726-74399493b081" path="/var/lib/kubelet/pods/05feb1b9-2478-4af3-8726-74399493b081/volumes" Jul 14 20:30:51.257140 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:30:51.257105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:31:16.251268 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.251231 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:31:16.251866 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.251609 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05feb1b9-2478-4af3-8726-74399493b081" containerName="main" Jul 14 20:31:16.251866 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.251621 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="05feb1b9-2478-4af3-8726-74399493b081" containerName="main" Jul 14 20:31:16.251866 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.251687 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="05feb1b9-2478-4af3-8726-74399493b081" containerName="main" Jul 14 20:31:16.254284 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.254262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.257812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.257789 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"default-dockercfg-ndhvd\"" Jul 14 20:31:16.257812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.257804 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"sub-test-model-d04b66-kserve-self-signed-certs\"" Jul 14 20:31:16.257986 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.257822 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"kube-root-ca.crt\"" Jul 14 20:31:16.257986 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.257804 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"openshift-service-ca.crt\"" Jul 14 20:31:16.263558 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.263537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:31:16.380762 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.380939 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.380939 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.380939 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.380939 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.380939 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.380897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxk2k\" (UniqueName: \"kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482149 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482352 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482352 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482352 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482352 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482352 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxk2k\" (UniqueName: \"kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482716 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482778 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.482822 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.482771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.484644 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.484623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.484863 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.484845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.492346 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.492319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxk2k\" (UniqueName: \"kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k\") pod \"sub-test-model-d04b66-kserve-865dc6d595-fswfn\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.565152 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.565073 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:16.698041 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:16.698011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:31:16.699483 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:31:16.699459 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc1903cf_a435_41ea_8460_d060c38a50ea.slice/crio-98ab934e74ac0bfb242b377a8fd129908773f3aaf9d4fbdaef4b8f6dc0650c5d WatchSource:0}: Error finding container 98ab934e74ac0bfb242b377a8fd129908773f3aaf9d4fbdaef4b8f6dc0650c5d: Status 404 returned error can't find the container with id 98ab934e74ac0bfb242b377a8fd129908773f3aaf9d4fbdaef4b8f6dc0650c5d Jul 14 20:31:17.086263 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:17.086228 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" event={"ID":"dc1903cf-a435-41ea-8460-d060c38a50ea","Type":"ContainerStarted","Data":"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0"} Jul 14 20:31:17.086263 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:17.086270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" event={"ID":"dc1903cf-a435-41ea-8460-d060c38a50ea","Type":"ContainerStarted","Data":"98ab934e74ac0bfb242b377a8fd129908773f3aaf9d4fbdaef4b8f6dc0650c5d"} Jul 14 20:31:17.105710 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:17.105660 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" podStartSLOduration=1.105643698 podStartE2EDuration="1.105643698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:31:17.10304255 +0000 UTC m=+2626.947379436" watchObservedRunningTime="2026-07-14 20:31:17.105643698 +0000 UTC m=+2626.949980573" Jul 14 20:31:20.756444 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:20.756409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:31:26.565913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:26.565875 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:26.565913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:26.565924 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:26.584376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:26.584342 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:27.151594 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:27.151556 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:31:35.361879 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:35.361844 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:31:50.542050 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:50.542014 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:31:54.948145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:31:54.948109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:32:05.562518 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.562445 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" podUID="dc1903cf-a435-41ea-8460-d060c38a50ea" containerName="main" containerID="cri-o://d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0" gracePeriod=45 Jul 14 20:32:05.820867 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.820794 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:32:05.933174 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933141 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933181 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933204 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933327 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxk2k\" (UniqueName: \"kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933369 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933463 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:05.933603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933474 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home" (OuterVolumeSpecName: "home") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:05.933603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm\") pod \"dc1903cf-a435-41ea-8460-d060c38a50ea\" (UID: \"dc1903cf-a435-41ea-8460-d060c38a50ea\") " Jul 14 20:32:05.933798 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933691 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:05.933913 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933895 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-model-cache\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:05.933965 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933919 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-tmp-dir\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:05.933965 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.933931 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-home\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:05.935648 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.935626 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm" (OuterVolumeSpecName: "dshm") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:05.935744 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.935724 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:32:05.935799 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:05.935779 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k" (OuterVolumeSpecName: "kube-api-access-vxk2k") pod "dc1903cf-a435-41ea-8460-d060c38a50ea" (UID: "dc1903cf-a435-41ea-8460-d060c38a50ea"). InnerVolumeSpecName "kube-api-access-vxk2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:06.034660 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.034626 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxk2k\" (UniqueName: \"kubernetes.io/projected/dc1903cf-a435-41ea-8460-d060c38a50ea-kube-api-access-vxk2k\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:06.034660 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.034653 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc1903cf-a435-41ea-8460-d060c38a50ea-dshm\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:06.034660 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.034661 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc1903cf-a435-41ea-8460-d060c38a50ea-tls-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:32:06.347711 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.347671 2574 generic.go:356] "Generic (PLEG): container finished" podID="dc1903cf-a435-41ea-8460-d060c38a50ea" containerID="d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0" exitCode=0 Jul 14 20:32:06.347711 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.347716 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" Jul 14 20:32:06.347923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.347729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" event={"ID":"dc1903cf-a435-41ea-8460-d060c38a50ea","Type":"ContainerDied","Data":"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0"} Jul 14 20:32:06.347923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.347769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn" event={"ID":"dc1903cf-a435-41ea-8460-d060c38a50ea","Type":"ContainerDied","Data":"98ab934e74ac0bfb242b377a8fd129908773f3aaf9d4fbdaef4b8f6dc0650c5d"} Jul 14 20:32:06.347923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.347792 2574 scope.go:117] "RemoveContainer" containerID="d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0" Jul 14 20:32:06.359364 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.358879 2574 scope.go:117] "RemoveContainer" containerID="d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0" Jul 14 20:32:06.359364 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:32:06.359232 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0\": container with ID starting with d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0 not found: ID does not exist" containerID="d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0" Jul 14 20:32:06.359364 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.359273 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0"} err="failed to get container status \"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0\": rpc error: code = NotFound desc = could not find container \"d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0\": container with ID starting with d2caba2ca6fbb1acca5f2e54d7b8f82ac5d74aed1cc5aee016b3dacd0c406db0 not found: ID does not exist" Jul 14 20:32:06.375187 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.375163 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:32:06.379392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.379370 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/sub-test-model-d04b66-kserve-865dc6d595-fswfn"] Jul 14 20:32:06.600361 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:06.600281 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1903cf-a435-41ea-8460-d060c38a50ea" path="/var/lib/kubelet/pods/dc1903cf-a435-41ea-8460-d060c38a50ea/volumes" Jul 14 20:32:11.056793 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:11.056763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:32:26.049999 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.049919 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:32:26.050554 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.050449 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1903cf-a435-41ea-8460-d060c38a50ea" containerName="main" Jul 14 20:32:26.050554 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.050469 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1903cf-a435-41ea-8460-d060c38a50ea" containerName="main" Jul 14 20:32:26.050689 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.050572 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1903cf-a435-41ea-8460-d060c38a50ea" containerName="main" Jul 14 20:32:26.053804 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.053781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.057272 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.057246 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"openshift-service-ca.crt\"" Jul 14 20:32:26.057521 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.057247 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"default-dockercfg-ndhvd\"" Jul 14 20:32:26.057715 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.057287 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"kube-root-ca.crt\"" Jul 14 20:32:26.057927 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.057321 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d04b66\"/\"rate-test-model-d04b66-kserve-self-signed-certs\"" Jul 14 20:32:26.060893 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.060868 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:32:26.216102 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.216295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.216295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.216295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.216295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.216295 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.216275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnjpz\" (UniqueName: \"kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjpz\" (UniqueName: \"kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317892 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317892 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.317892 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.318019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.318019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.317991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.318086 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.318018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.320173 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.320147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.320410 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.320390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.325663 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.325641 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjpz\" (UniqueName: \"kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz\") pod \"rate-test-model-d04b66-kserve-7b5f46fd75-8sztz\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.365534 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.365506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:26.507927 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:26.507899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:32:26.509558 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:32:26.509483 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a6ddf8d_a9b9_49e4_aa05_09eb7ddd5481.slice/crio-5458b5aedf7187e7e5cc2ca234afda3508c120ed03e4893e6f265517252a52ad WatchSource:0}: Error finding container 5458b5aedf7187e7e5cc2ca234afda3508c120ed03e4893e6f265517252a52ad: Status 404 returned error can't find the container with id 5458b5aedf7187e7e5cc2ca234afda3508c120ed03e4893e6f265517252a52ad Jul 14 20:32:27.467857 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:27.467811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" event={"ID":"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481","Type":"ContainerStarted","Data":"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa"} Jul 14 20:32:27.468239 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:27.467868 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" event={"ID":"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481","Type":"ContainerStarted","Data":"5458b5aedf7187e7e5cc2ca234afda3508c120ed03e4893e6f265517252a52ad"} Jul 14 20:32:27.484917 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:27.484866 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" podStartSLOduration=1.48485194 podStartE2EDuration="1.48485194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:32:27.48342609 +0000 UTC m=+2697.327762967" watchObservedRunningTime="2026-07-14 20:32:27.48485194 +0000 UTC m=+2697.329188847" Jul 14 20:32:30.752940 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:30.752909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:32:36.366396 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:36.366361 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:36.366825 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:36.366412 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:36.378854 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:36.378829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:36.529014 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:36.528987 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:32:45.052683 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:32:45.052646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:33:04.760568 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:04.760521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:33:09.366735 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:09.366704 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:33:13.449233 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.449195 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066908848"] Jul 14 20:33:13.452579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.452562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:13.455052 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.455028 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2ln2\"" Jul 14 20:33:13.456737 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.456714 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066908848"] Jul 14 20:33:13.543500 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.543452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh94v\" (UniqueName: \"kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v\") pod \"test-curl-7704-140546066908848\" (UID: \"10c8e583-3b79-4077-b297-fa37bcba5c40\") " pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:13.644159 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.644122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh94v\" (UniqueName: \"kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v\") pod \"test-curl-7704-140546066908848\" (UID: \"10c8e583-3b79-4077-b297-fa37bcba5c40\") " pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:13.652321 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.652297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh94v\" (UniqueName: \"kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v\") pod \"test-curl-7704-140546066908848\" (UID: \"10c8e583-3b79-4077-b297-fa37bcba5c40\") " pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:13.762382 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.762298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:13.905679 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.905648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066908848"] Jul 14 20:33:13.907038 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:33:13.907010 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10c8e583_3b79_4077_b297_fa37bcba5c40.slice/crio-3c30583f97fa6a2992f5e3920d3f1be63016c3c2c511c71acc295ab57cba33c2 WatchSource:0}: Error finding container 3c30583f97fa6a2992f5e3920d3f1be63016c3c2c511c71acc295ab57cba33c2: Status 404 returned error can't find the container with id 3c30583f97fa6a2992f5e3920d3f1be63016c3c2c511c71acc295ab57cba33c2 Jul 14 20:33:13.908632 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:13.908614 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:33:14.721550 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:14.721479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066908848" event={"ID":"10c8e583-3b79-4077-b297-fa37bcba5c40","Type":"ContainerStarted","Data":"3a77449bd03386b575735005fe0274b32d2fc827d6f66bdbb00452daf02b9933"} Jul 14 20:33:14.721944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:14.721566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066908848" event={"ID":"10c8e583-3b79-4077-b297-fa37bcba5c40","Type":"ContainerStarted","Data":"3c30583f97fa6a2992f5e3920d3f1be63016c3c2c511c71acc295ab57cba33c2"} Jul 14 20:33:14.848117 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:14.848083 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066908848_10c8e583-3b79-4077-b297-fa37bcba5c40/test-curl-7704-140546066908848/0.log" Jul 14 20:33:15.728621 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:15.728580 2574 generic.go:356] "Generic (PLEG): container finished" podID="10c8e583-3b79-4077-b297-fa37bcba5c40" containerID="3a77449bd03386b575735005fe0274b32d2fc827d6f66bdbb00452daf02b9933" exitCode=0 Jul 14 20:33:15.729021 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:15.728663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066908848" event={"ID":"10c8e583-3b79-4077-b297-fa37bcba5c40","Type":"ContainerDied","Data":"3a77449bd03386b575735005fe0274b32d2fc827d6f66bdbb00452daf02b9933"} Jul 14 20:33:16.867803 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:16.867782 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:16.899272 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:16.899241 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066908848"] Jul 14 20:33:16.903443 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:16.903412 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066908848"] Jul 14 20:33:16.977097 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:16.977070 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh94v\" (UniqueName: \"kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v\") pod \"10c8e583-3b79-4077-b297-fa37bcba5c40\" (UID: \"10c8e583-3b79-4077-b297-fa37bcba5c40\") " Jul 14 20:33:16.979470 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:16.979437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v" (OuterVolumeSpecName: "kube-api-access-xh94v") pod "10c8e583-3b79-4077-b297-fa37bcba5c40" (UID: "10c8e583-3b79-4077-b297-fa37bcba5c40"). InnerVolumeSpecName "kube-api-access-xh94v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:17.078702 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:17.078677 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh94v\" (UniqueName: \"kubernetes.io/projected/10c8e583-3b79-4077-b297-fa37bcba5c40-kube-api-access-xh94v\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:17.740597 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:17.740567 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:17.740772 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:17.740570 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c30583f97fa6a2992f5e3920d3f1be63016c3c2c511c71acc295ab57cba33c2" Jul 14 20:33:17.750036 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:17.750006 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-140546066908848\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="10c8e583-3b79-4077-b297-fa37bcba5c40" pod="opendatahub/test-curl-7704-140546066908848" Jul 14 20:33:18.600161 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:18.600130 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c8e583-3b79-4077-b297-fa37bcba5c40" path="/var/lib/kubelet/pods/10c8e583-3b79-4077-b297-fa37bcba5c40/volumes" Jul 14 20:33:19.251442 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.251408 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:33:19.476174 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.476139 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066850560"] Jul 14 20:33:19.476552 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.476537 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c8e583-3b79-4077-b297-fa37bcba5c40" containerName="test-curl-7704-140546066908848" Jul 14 20:33:19.476624 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.476554 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c8e583-3b79-4077-b297-fa37bcba5c40" containerName="test-curl-7704-140546066908848" Jul 14 20:33:19.476668 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.476633 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c8e583-3b79-4077-b297-fa37bcba5c40" containerName="test-curl-7704-140546066908848" Jul 14 20:33:19.479972 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.479956 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:19.482111 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.482092 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2ln2\"" Jul 14 20:33:19.485647 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.485622 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"] Jul 14 20:33:19.601922 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.601821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd8jc\" (UniqueName: \"kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc\") pod \"test-curl-7704-140546066850560\" (UID: \"3af72546-0149-4d7b-8583-717f098720ec\") " pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:19.702701 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.702662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd8jc\" (UniqueName: \"kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc\") pod \"test-curl-7704-140546066850560\" (UID: \"3af72546-0149-4d7b-8583-717f098720ec\") " pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:19.710988 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.710946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd8jc\" (UniqueName: \"kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc\") pod \"test-curl-7704-140546066850560\" (UID: \"3af72546-0149-4d7b-8583-717f098720ec\") " pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:19.790227 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.790198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:19.930713 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:19.930687 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"] Jul 14 20:33:19.932673 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:33:19.932635 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af72546_0149_4d7b_8583_717f098720ec.slice/crio-3d8a40e90f4e4381cbbda5988a8889a3500609fa856f7426304157f3a11af404 WatchSource:0}: Error finding container 3d8a40e90f4e4381cbbda5988a8889a3500609fa856f7426304157f3a11af404: Status 404 returned error can't find the container with id 3d8a40e90f4e4381cbbda5988a8889a3500609fa856f7426304157f3a11af404 Jul 14 20:33:20.765616 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:20.765580 2574 generic.go:356] "Generic (PLEG): container finished" podID="3af72546-0149-4d7b-8583-717f098720ec" containerID="db387b3e9e3667356b67440310929e2a22b8c815a29be74fa992728b85b03bf2" exitCode=0 Jul 14 20:33:20.765962 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:20.765664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066850560" event={"ID":"3af72546-0149-4d7b-8583-717f098720ec","Type":"ContainerDied","Data":"db387b3e9e3667356b67440310929e2a22b8c815a29be74fa992728b85b03bf2"} Jul 14 20:33:20.765962 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:20.765710 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066850560" event={"ID":"3af72546-0149-4d7b-8583-717f098720ec","Type":"ContainerStarted","Data":"3d8a40e90f4e4381cbbda5988a8889a3500609fa856f7426304157f3a11af404"} Jul 14 20:33:21.916994 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:21.916973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:21.938220 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:21.938186 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066850560_3af72546-0149-4d7b-8583-717f098720ec/test-curl-7704-140546066850560/0.log" Jul 14 20:33:21.962473 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:21.962421 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"] Jul 14 20:33:21.964573 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:21.964554 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066850560"] Jul 14 20:33:22.023642 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.023584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd8jc\" (UniqueName: \"kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc\") pod \"3af72546-0149-4d7b-8583-717f098720ec\" (UID: \"3af72546-0149-4d7b-8583-717f098720ec\") " Jul 14 20:33:22.025781 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.025738 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc" (OuterVolumeSpecName: "kube-api-access-vd8jc") pod "3af72546-0149-4d7b-8583-717f098720ec" (UID: "3af72546-0149-4d7b-8583-717f098720ec"). InnerVolumeSpecName "kube-api-access-vd8jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:22.094775 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.094748 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7704-140546066851568"] Jul 14 20:33:22.095097 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.095084 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af72546-0149-4d7b-8583-717f098720ec" containerName="test-curl-7704-140546066850560" Jul 14 20:33:22.095140 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.095099 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af72546-0149-4d7b-8583-717f098720ec" containerName="test-curl-7704-140546066850560" Jul 14 20:33:22.095175 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.095156 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af72546-0149-4d7b-8583-717f098720ec" containerName="test-curl-7704-140546066850560" Jul 14 20:33:22.098021 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.098006 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:22.104592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.104561 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"] Jul 14 20:33:22.125167 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.125141 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd8jc\" (UniqueName: \"kubernetes.io/projected/3af72546-0149-4d7b-8583-717f098720ec-kube-api-access-vd8jc\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:22.225995 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.225958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckr7\" (UniqueName: \"kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7\") pod \"test-curl-7704-140546066851568\" (UID: \"1a5f85a7-5732-4f44-aa0d-6f41d1e45580\") " pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:22.326562 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.326460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckr7\" (UniqueName: \"kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7\") pod \"test-curl-7704-140546066851568\" (UID: \"1a5f85a7-5732-4f44-aa0d-6f41d1e45580\") " pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:22.334603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.334574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckr7\" (UniqueName: \"kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7\") pod \"test-curl-7704-140546066851568\" (UID: \"1a5f85a7-5732-4f44-aa0d-6f41d1e45580\") " pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:22.408535 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.408479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:22.543857 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.543829 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"] Jul 14 20:33:22.545359 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:33:22.545331 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5f85a7_5732_4f44_aa0d_6f41d1e45580.slice/crio-d040ab6fb66321364b6ea75b26ea3e2c415d25ea022959903d4bef58cdde2260 WatchSource:0}: Error finding container d040ab6fb66321364b6ea75b26ea3e2c415d25ea022959903d4bef58cdde2260: Status 404 returned error can't find the container with id d040ab6fb66321364b6ea75b26ea3e2c415d25ea022959903d4bef58cdde2260 Jul 14 20:33:22.605433 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.605399 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af72546-0149-4d7b-8583-717f098720ec" path="/var/lib/kubelet/pods/3af72546-0149-4d7b-8583-717f098720ec/volumes" Jul 14 20:33:22.777183 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.777152 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066851568" event={"ID":"1a5f85a7-5732-4f44-aa0d-6f41d1e45580","Type":"ContainerStarted","Data":"d040ab6fb66321364b6ea75b26ea3e2c415d25ea022959903d4bef58cdde2260"} Jul 14 20:33:22.778305 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.778278 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066850560" Jul 14 20:33:22.778422 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:22.778313 2574 scope.go:117] "RemoveContainer" containerID="db387b3e9e3667356b67440310929e2a22b8c815a29be74fa992728b85b03bf2" Jul 14 20:33:23.785079 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:23.785041 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" containerID="1701ad7acc8965167ca620a36efb3c9ad775351d63687f0277cb2c2cbe4d6dc5" exitCode=0 Jul 14 20:33:23.785514 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:23.785121 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7704-140546066851568" event={"ID":"1a5f85a7-5732-4f44-aa0d-6f41d1e45580","Type":"ContainerDied","Data":"1701ad7acc8965167ca620a36efb3c9ad775351d63687f0277cb2c2cbe4d6dc5"} Jul 14 20:33:24.390453 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.390408 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" podUID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" containerName="main" containerID="cri-o://8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa" gracePeriod=45 Jul 14 20:33:24.640852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.640791 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:33:24.747692 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747663 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.747872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747736 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.747872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747756 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.747872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747785 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.747872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnjpz\" (UniqueName: \"kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.748073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747896 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm\") pod \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\" (UID: \"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481\") " Jul 14 20:33:24.748073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.747956 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:24.748073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.748001 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:24.748073 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.748028 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home" (OuterVolumeSpecName: "home") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:24.748283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.748182 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-model-cache\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.748283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.748218 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tmp-dir\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.748283 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.748228 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-home\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.750087 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.750055 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm" (OuterVolumeSpecName: "dshm") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:24.750194 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.750094 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:33:24.750267 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.750187 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz" (OuterVolumeSpecName: "kube-api-access-mnjpz") pod "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" (UID: "3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481"). InnerVolumeSpecName "kube-api-access-mnjpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:24.793361 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.793332 2574 generic.go:356] "Generic (PLEG): container finished" podID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" containerID="8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa" exitCode=0 Jul 14 20:33:24.793729 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.793368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" event={"ID":"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481","Type":"ContainerDied","Data":"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa"} Jul 14 20:33:24.793729 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.793419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" event={"ID":"3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481","Type":"ContainerDied","Data":"5458b5aedf7187e7e5cc2ca234afda3508c120ed03e4893e6f265517252a52ad"} Jul 14 20:33:24.793729 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.793438 2574 scope.go:117] "RemoveContainer" containerID="8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa" Jul 14 20:33:24.793729 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.793383 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz" Jul 14 20:33:24.803193 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.803175 2574 scope.go:117] "RemoveContainer" containerID="8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa" Jul 14 20:33:24.803450 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:24.803432 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa\": container with ID starting with 8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa not found: ID does not exist" containerID="8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa" Jul 14 20:33:24.803553 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.803461 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa"} err="failed to get container status \"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa\": rpc error: code = NotFound desc = could not find container \"8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa\": container with ID starting with 8eb894ebc00ff3ee80ee20e74957f42d5cd886c63a6f4467c8b18c80ab11a8aa not found: ID does not exist" Jul 14 20:33:24.818084 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.818053 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:33:24.822019 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.821988 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d04b66/rate-test-model-d04b66-kserve-7b5f46fd75-8sztz"] Jul 14 20:33:24.849615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.849478 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-tls-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.849615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.849542 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnjpz\" (UniqueName: \"kubernetes.io/projected/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-kube-api-access-mnjpz\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.849615 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.849556 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481-dshm\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:24.916593 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.916566 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:24.936409 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.936375 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7704-140546066851568_1a5f85a7-5732-4f44-aa0d-6f41d1e45580/test-curl-7704-140546066851568/0.log" Jul 14 20:33:24.962404 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.962377 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"] Jul 14 20:33:24.966090 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:24.966069 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7704-140546066851568"] Jul 14 20:33:25.051320 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:25.051296 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ckr7\" (UniqueName: \"kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7\") pod \"1a5f85a7-5732-4f44-aa0d-6f41d1e45580\" (UID: \"1a5f85a7-5732-4f44-aa0d-6f41d1e45580\") " Jul 14 20:33:25.053421 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:25.053400 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7" (OuterVolumeSpecName: "kube-api-access-5ckr7") pod "1a5f85a7-5732-4f44-aa0d-6f41d1e45580" (UID: "1a5f85a7-5732-4f44-aa0d-6f41d1e45580"). InnerVolumeSpecName "kube-api-access-5ckr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:25.152915 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:25.152886 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ckr7\" (UniqueName: \"kubernetes.io/projected/1a5f85a7-5732-4f44-aa0d-6f41d1e45580-kube-api-access-5ckr7\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:25.801245 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:25.801221 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:25.801740 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:25.801224 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d040ab6fb66321364b6ea75b26ea3e2c415d25ea022959903d4bef58cdde2260" Jul 14 20:33:25.810723 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:25.810693 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7704-140546066851568\" is forbidden: User \"system:node:ip-10-0-129-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-28.ec2.internal' and this object" podUID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" pod="opendatahub/test-curl-7704-140546066851568" Jul 14 20:33:26.600468 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:26.600430 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" path="/var/lib/kubelet/pods/1a5f85a7-5732-4f44-aa0d-6f41d1e45580/volumes" Jul 14 20:33:26.600990 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:26.600966 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" path="/var/lib/kubelet/pods/3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481/volumes" Jul 14 20:33:42.155704 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:42.155670 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:33:53.388290 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:53.388209 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:33:53.388714 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:53.388429 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" containerID="cri-o://025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00" gracePeriod=30 Jul 14 20:33:58.644631 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.644607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:33:58.739632 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739600 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4rdg\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.739812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739690 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.739812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739718 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.739812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.739812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739768 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.739812 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.740106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.740106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.740106 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.739951 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data\") pod \"bd39e3a4-90c3-4805-8fec-bfda27a48398\" (UID: \"bd39e3a4-90c3-4805-8fec-bfda27a48398\") " Jul 14 20:33:58.740653 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.740380 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:58.740653 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.740584 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:58.740653 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.740602 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:58.740895 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.740716 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:33:58.741297 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.741265 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:58.742343 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.742315 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg" (OuterVolumeSpecName: "kube-api-access-s4rdg") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "kube-api-access-s4rdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:58.742843 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.742816 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:33:58.743036 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.743016 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:33:58.743125 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.743108 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd39e3a4-90c3-4805-8fec-bfda27a48398" (UID: "bd39e3a4-90c3-4805-8fec-bfda27a48398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:33:58.841393 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841363 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841393 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841390 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841400 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841407 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4rdg\" (UniqueName: \"kubernetes.io/projected/bd39e3a4-90c3-4805-8fec-bfda27a48398-kube-api-access-s4rdg\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841416 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841425 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841432 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd39e3a4-90c3-4805-8fec-bfda27a48398-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841439 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.841587 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.841447 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd39e3a4-90c3-4805-8fec-bfda27a48398-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:33:58.982406 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.982373 2574 generic.go:356] "Generic (PLEG): container finished" podID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerID="025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00" exitCode=0 Jul 14 20:33:58.982592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.982423 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" Jul 14 20:33:58.982592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.982451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerDied","Data":"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00"} Jul 14 20:33:58.982592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.982540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t" event={"ID":"bd39e3a4-90c3-4805-8fec-bfda27a48398","Type":"ContainerDied","Data":"4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28"} Jul 14 20:33:58.982592 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.982562 2574 scope.go:117] "RemoveContainer" containerID="025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00" Jul 14 20:33:58.992289 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:58.992272 2574 scope.go:117] "RemoveContainer" containerID="db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a" Jul 14 20:33:59.000666 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.000647 2574 scope.go:117] "RemoveContainer" containerID="025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00" Jul 14 20:33:59.000915 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:59.000895 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00\": container with ID starting with 025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00 not found: ID does not exist" containerID="025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00" Jul 14 20:33:59.000988 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.000928 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00"} err="failed to get container status \"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00\": rpc error: code = NotFound desc = could not find container \"025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00\": container with ID starting with 025e12f4bba5d8621b98a1387b5b18b94193840026d8b59ccf8bc7bd08ba8f00 not found: ID does not exist" Jul 14 20:33:59.000988 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.000960 2574 scope.go:117] "RemoveContainer" containerID="db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a" Jul 14 20:33:59.001189 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:59.001173 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a\": container with ID starting with db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a not found: ID does not exist" containerID="db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a" Jul 14 20:33:59.001248 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.001196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a"} err="failed to get container status \"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a\": rpc error: code = NotFound desc = could not find container \"db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a\": container with ID starting with db05a4f7e546fffa7006e3882215cbe2f0843e4fcb1791784b1cee007b50de3a not found: ID does not exist" Jul 14 20:33:59.006327 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.006306 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:33:59.009382 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:33:59.009360 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39e3a4_90c3_4805_8fec_bfda27a48398.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd39e3a4_90c3_4805_8fec_bfda27a48398.slice/crio-4403ead7d97fe7ebab55aaad49c7c490d3051f0920ee09ec14d08ad1e60f5f28\": RecentStats: unable to find data in memory cache]" Jul 14 20:33:59.010127 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:33:59.010108 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-10ef4d-openshift-default-6f4599fddc-scf8t"] Jul 14 20:34:00.599420 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:00.599389 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" path="/var/lib/kubelet/pods/bd39e3a4-90c3-4805-8fec-bfda27a48398/volumes" Jul 14 20:34:07.160945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:07.160898 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-82b2j"] Jul 14 20:34:19.589845 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:19.589809 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:34:19.590392 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:19.590104 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" containerID="cri-o://240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2" gracePeriod=30 Jul 14 20:34:20.310852 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:20.310816 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d77bf8dc-6jskd_c02dcb30-b969-4fdc-9062-ec72db8d77ef/authorino/0.log" Jul 14 20:34:24.849784 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.849760 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:34:24.864251 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864201 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqm9f\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864251 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864233 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864254 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864287 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864314 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864368 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864640 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864640 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864524 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864640 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo\") pod \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\" (UID: \"791d212c-2ac2-4a98-ac8a-1c05f7b4db69\") " Jul 14 20:34:24.864796 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864653 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:34:24.864796 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864684 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:34:24.864944 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864923 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-credential-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.865003 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864952 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-certs\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.865003 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.864926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:34:24.865149 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.865115 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:34:24.865507 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.865468 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data" (OuterVolumeSpecName: "istio-data") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:34:24.866528 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.866485 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:34:24.866820 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.866785 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token" (OuterVolumeSpecName: "istio-token") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:34:24.867076 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.867056 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f" (OuterVolumeSpecName: "kube-api-access-rqm9f") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "kube-api-access-rqm9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:34:24.867151 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.867081 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "791d212c-2ac2-4a98-ac8a-1c05f7b4db69" (UID: "791d212c-2ac2-4a98-ac8a-1c05f7b4db69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:34:24.965753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965716 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm9f\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-kube-api-access-rqm9f\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965752 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-envoy\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965761 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-data\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965753 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965769 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-token\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965777 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istiod-ca-cert\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965785 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-workload-socket\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:24.965964 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:24.965792 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/791d212c-2ac2-4a98-ac8a-1c05f7b4db69-istio-podinfo\") on node \"ip-10-0-129-28.ec2.internal\" DevicePath \"\"" Jul 14 20:34:25.128569 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.128456 2574 generic.go:356] "Generic (PLEG): container finished" podID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerID="240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2" exitCode=0 Jul 14 20:34:25.128569 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.128547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerDied","Data":"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2"} Jul 14 20:34:25.128569 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.128560 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" Jul 14 20:34:25.128815 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.128633 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh" event={"ID":"791d212c-2ac2-4a98-ac8a-1c05f7b4db69","Type":"ContainerDied","Data":"09bb2a3022d24814ff20013858881c74376b1783e2069e6db7cb1e9dbc42c63f"} Jul 14 20:34:25.128815 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.128659 2574 scope.go:117] "RemoveContainer" containerID="240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2" Jul 14 20:34:25.138149 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.138132 2574 scope.go:117] "RemoveContainer" containerID="f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9" Jul 14 20:34:25.146530 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.146510 2574 scope.go:117] "RemoveContainer" containerID="240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2" Jul 14 20:34:25.146820 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:34:25.146801 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2\": container with ID starting with 240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2 not found: ID does not exist" containerID="240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2" Jul 14 20:34:25.146872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.146830 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2"} err="failed to get container status \"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2\": rpc error: code = NotFound desc = could not find container \"240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2\": container with ID starting with 240da48b28a86f944591c5f610c8f98812fce026a492890b6eac7c57431331b2 not found: ID does not exist" Jul 14 20:34:25.146872 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.146848 2574 scope.go:117] "RemoveContainer" containerID="f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9" Jul 14 20:34:25.147076 ip-10-0-129-28 kubenswrapper[2574]: E0714 20:34:25.147059 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9\": container with ID starting with f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9 not found: ID does not exist" containerID="f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9" Jul 14 20:34:25.147123 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.147082 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9"} err="failed to get container status \"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9\": rpc error: code = NotFound desc = could not find container \"f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9\": container with ID starting with f92afe9486cf079f9a5f3b0d83d3210f794df071914840a4266c70c09eb225a9 not found: ID does not exist" Jul 14 20:34:25.160264 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.160241 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:34:25.162790 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:25.162771 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d04b66-openshift-default-5588fb6499-9smhh"] Jul 14 20:34:26.056945 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:26.056908 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5b6jd_07ae2cb9-a573-41ef-b100-3f643925c6c4/manager/0.log" Jul 14 20:34:26.329927 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:26.329850 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-gk7lj_0894dd72-2728-437f-a9de-eb6d75738a92/copy-manifests/0.log" Jul 14 20:34:26.338277 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:26.338255 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-gk7lj_0894dd72-2728-437f-a9de-eb6d75738a92/manager/0.log" Jul 14 20:34:26.601315 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:26.601237 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" path="/var/lib/kubelet/pods/791d212c-2ac2-4a98-ac8a-1c05f7b4db69/volumes" Jul 14 20:34:29.629559 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.629521 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/util/0.log" Jul 14 20:34:29.636892 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.636866 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/pull/0.log" Jul 14 20:34:29.644251 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.644219 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/extract/0.log" Jul 14 20:34:29.768714 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.768686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/util/0.log" Jul 14 20:34:29.776544 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.776518 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/pull/0.log" Jul 14 20:34:29.784690 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.784662 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/extract/0.log" Jul 14 20:34:29.910894 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.910839 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/util/0.log" Jul 14 20:34:29.918517 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.918480 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/pull/0.log" Jul 14 20:34:29.925589 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:29.925572 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/extract/0.log" Jul 14 20:34:30.049814 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.049798 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/extract/0.log" Jul 14 20:34:30.057089 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.057066 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/util/0.log" Jul 14 20:34:30.064011 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.063981 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/pull/0.log" Jul 14 20:34:30.197716 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.197670 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d77bf8dc-6jskd_c02dcb30-b969-4fdc-9062-ec72db8d77ef/authorino/0.log" Jul 14 20:34:30.322334 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.322303 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zkq2q_647410ee-9e16-4eed-a54c-27421157a0d4/manager/0.log" Jul 14 20:34:30.692702 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.692673 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dr9b6_a45b28b9-2855-4a5b-a776-fea8c7069c98/registry-server/0.log" Jul 14 20:34:30.825923 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.825896 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-whsp4_eda53b00-ca4e-4984-a2d9-c902960fcab7/manager/0.log" Jul 14 20:34:30.946175 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:30.946153 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-82b2j_ce3de322-d2c2-455c-8206-c8f2a611a607/limitador/0.log" Jul 14 20:34:31.452356 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:31.452322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67884mdb_250b0041-6ff0-4dff-b146-b2e02e122934/istio-proxy/0.log" Jul 14 20:34:31.971020 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:31.970991 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/1.log" Jul 14 20:34:37.207182 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207146 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tnh4v/must-gather-5282b"] Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207500 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207511 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207531 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207537 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207545 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207550 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207557 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" containerName="test-curl-7704-140546066851568" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" containerName="test-curl-7704-140546066851568" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207579 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207579 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207586 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" containerName="main" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207591 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" containerName="main" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207661 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207669 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207675 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="791d212c-2ac2-4a98-ac8a-1c05f7b4db69" containerName="istio-proxy" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207683 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd39e3a4-90c3-4805-8fec-bfda27a48398" containerName="istio-proxy" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207688 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a5f85a7-5732-4f44-aa0d-6f41d1e45580" containerName="test-curl-7704-140546066851568" Jul 14 20:34:37.207906 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.207693 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6ddf8d-a9b9-49e4-aa05-09eb7ddd5481" containerName="main" Jul 14 20:34:37.212125 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.212109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.214739 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.214718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnh4v\"/\"kube-root-ca.crt\"" Jul 14 20:34:37.215632 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.215618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnh4v\"/\"openshift-service-ca.crt\"" Jul 14 20:34:37.215680 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.215637 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tnh4v\"/\"default-dockercfg-7bx5v\"" Jul 14 20:34:37.221928 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.221900 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/must-gather-5282b"] Jul 14 20:34:37.273901 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.273866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrxc\" (UniqueName: \"kubernetes.io/projected/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-kube-api-access-grrxc\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.274078 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.273935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-must-gather-output\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.375350 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.375318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrxc\" (UniqueName: \"kubernetes.io/projected/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-kube-api-access-grrxc\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.375543 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.375377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-must-gather-output\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.375739 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.375720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-must-gather-output\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.383846 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.383823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrxc\" (UniqueName: \"kubernetes.io/projected/3a47d669-b429-4aa9-a6b5-9a9aafe6b23f-kube-api-access-grrxc\") pod \"must-gather-5282b\" (UID: \"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f\") " pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.523186 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.523118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/must-gather-5282b" Jul 14 20:34:37.672082 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:37.672058 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/must-gather-5282b"] Jul 14 20:34:37.674663 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:34:37.674635 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a47d669_b429_4aa9_a6b5_9a9aafe6b23f.slice/crio-080882fcc4953a5179ced20542c8758002ef3a6b2938e7fda4eaa0e0f2050a4b WatchSource:0}: Error finding container 080882fcc4953a5179ced20542c8758002ef3a6b2938e7fda4eaa0e0f2050a4b: Status 404 returned error can't find the container with id 080882fcc4953a5179ced20542c8758002ef3a6b2938e7fda4eaa0e0f2050a4b Jul 14 20:34:38.196562 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:38.196526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/must-gather-5282b" event={"ID":"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f","Type":"ContainerStarted","Data":"080882fcc4953a5179ced20542c8758002ef3a6b2938e7fda4eaa0e0f2050a4b"} Jul 14 20:34:39.207603 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:39.207560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/must-gather-5282b" event={"ID":"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f","Type":"ContainerStarted","Data":"a0c69f9dbcaffac4cf88955c25476e27c2f657e790ba198635f1e7eb2483987f"} Jul 14 20:34:39.208063 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:39.207619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/must-gather-5282b" event={"ID":"3a47d669-b429-4aa9-a6b5-9a9aafe6b23f","Type":"ContainerStarted","Data":"cbe22a8ca256e674638c5d9e8effb2e1512f12634499662f9db343f3be9cf789"} Jul 14 20:34:39.222597 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:39.222546 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tnh4v/must-gather-5282b" podStartSLOduration=1.3255930949999999 podStartE2EDuration="2.222526564s" totalImagesPullingTime="896.933469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:34:39.221663381 +0000 UTC m=+2829.066000256" watchObservedRunningTime="2026-07-14 20:34:39.222526564 +0000 UTC m=+2829.066863437" Jul 14 20:34:40.136714 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:40.136677 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8rvfr_daa50ea0-e19e-4c48-9afc-adc079abf2d5/global-pull-secret-syncer/0.log" Jul 14 20:34:40.305805 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:40.305773 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cpz46_a1881a13-f947-4dbd-b8cf-f89d4a860b61/konnectivity-agent/0.log" Jul 14 20:34:40.370685 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:40.370658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-28.ec2.internal_43ba1ebae5d7ee21a4751f58c11fde6b/haproxy/0.log" Jul 14 20:34:44.690651 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.690618 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/extract/0.log" Jul 14 20:34:44.719533 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.719471 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/util/0.log" Jul 14 20:34:44.745199 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.745166 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bl64z_7c96c264-6037-45c8-8727-7c9991b9fa17/pull/0.log" Jul 14 20:34:44.782372 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.782331 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/extract/0.log" Jul 14 20:34:44.814829 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.814800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/util/0.log" Jul 14 20:34:44.868584 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.868550 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hbn82_cbb8426a-e05d-4ddf-9fda-ed7df477b963/pull/0.log" Jul 14 20:34:44.937376 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.937317 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/extract/0.log" Jul 14 20:34:44.986625 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:44.986592 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/util/0.log" Jul 14 20:34:45.027332 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.027281 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73999gd_27fb17c4-864d-4661-b455-b2ed61c238f1/pull/0.log" Jul 14 20:34:45.069640 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.069610 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/extract/0.log" Jul 14 20:34:45.105178 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.105145 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/util/0.log" Jul 14 20:34:45.134371 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.134337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef16w4ws_baf1d59c-b932-4641-92e3-3ed3258fcfcf/pull/0.log" Jul 14 20:34:45.674037 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.673199 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78d77bf8dc-6jskd_c02dcb30-b969-4fdc-9062-ec72db8d77ef/authorino/0.log" Jul 14 20:34:45.715548 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.715440 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zkq2q_647410ee-9e16-4eed-a54c-27421157a0d4/manager/0.log" Jul 14 20:34:45.824281 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:45.824233 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dr9b6_a45b28b9-2855-4a5b-a776-fea8c7069c98/registry-server/0.log" Jul 14 20:34:46.042834 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:46.037390 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-whsp4_eda53b00-ca4e-4984-a2d9-c902960fcab7/manager/0.log" Jul 14 20:34:46.071197 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:46.070914 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-82b2j_ce3de322-d2c2-455c-8206-c8f2a611a607/limitador/0.log" Jul 14 20:34:48.154568 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.151430 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jhq4l_838f0de2-ba0f-4963-af68-e57a89cd2066/kube-state-metrics/0.log" Jul 14 20:34:48.178768 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.178736 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jhq4l_838f0de2-ba0f-4963-af68-e57a89cd2066/kube-rbac-proxy-main/0.log" Jul 14 20:34:48.203139 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.203110 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-jhq4l_838f0de2-ba0f-4963-af68-e57a89cd2066/kube-rbac-proxy-self/0.log" Jul 14 20:34:48.235956 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.235911 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b55f44bb6-fqpkh_9d2eb0ac-8b8f-429e-a36b-2f0c572d1a1d/metrics-server/0.log" Jul 14 20:34:48.263289 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.263262 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-7hdgb_09c295ef-6803-4875-927d-63c8e38a9a61/monitoring-plugin/0.log" Jul 14 20:34:48.319535 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.319477 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6wgd4_20e53405-752c-4424-8bd6-510cb766d7f9/node-exporter/0.log" Jul 14 20:34:48.343480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.343451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6wgd4_20e53405-752c-4424-8bd6-510cb766d7f9/kube-rbac-proxy/0.log" Jul 14 20:34:48.368437 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.368397 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6wgd4_20e53405-752c-4424-8bd6-510cb766d7f9/init-textfile/0.log" Jul 14 20:34:48.789619 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.789582 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xsss5_f2eef7cb-9465-4f2e-b48e-422f568529bf/prometheus-operator/0.log" Jul 14 20:34:48.808830 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.808800 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-xsss5_f2eef7cb-9465-4f2e-b48e-422f568529bf/kube-rbac-proxy/0.log" Jul 14 20:34:48.867039 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.867005 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bd7f5567c-k6x2b_1d09d6b1-222c-43a2-b4c8-22c49e746e71/telemeter-client/0.log" Jul 14 20:34:48.891721 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.891688 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bd7f5567c-k6x2b_1d09d6b1-222c-43a2-b4c8-22c49e746e71/reload/0.log" Jul 14 20:34:48.913477 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:48.913442 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7bd7f5567c-k6x2b_1d09d6b1-222c-43a2-b4c8-22c49e746e71/kube-rbac-proxy/0.log" Jul 14 20:34:50.511673 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.511625 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4"] Jul 14 20:34:50.520226 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.520199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.523372 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.522683 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4"] Jul 14 20:34:50.640845 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.640799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-sys\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.641179 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.641158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-proc\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.641386 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.641368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-lib-modules\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.641573 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.641557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr66b\" (UniqueName: \"kubernetes.io/projected/14f2a50e-462f-4727-9c99-f688ecd1bc30-kube-api-access-vr66b\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.641724 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.641710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-podres\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743025 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.742984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-sys\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743226 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-sys\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743306 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-proc\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-lib-modules\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743368 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr66b\" (UniqueName: \"kubernetes.io/projected/14f2a50e-462f-4727-9c99-f688ecd1bc30-kube-api-access-vr66b\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743480 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-podres\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743661 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-podres\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743840 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-proc\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.743958 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.743888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14f2a50e-462f-4727-9c99-f688ecd1bc30-lib-modules\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.752950 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.752926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr66b\" (UniqueName: \"kubernetes.io/projected/14f2a50e-462f-4727-9c99-f688ecd1bc30-kube-api-access-vr66b\") pod \"perf-node-gather-daemonset-pwlk4\" (UID: \"14f2a50e-462f-4727-9c99-f688ecd1bc30\") " pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:50.834774 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:50.834669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:51.010931 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:51.010903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4"] Jul 14 20:34:51.014434 ip-10-0-129-28 kubenswrapper[2574]: W0714 20:34:51.014405 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14f2a50e_462f_4727_9c99_f688ecd1bc30.slice/crio-9cb86091e449c481ac4a685564801b24f16b74ad77606d2f4f2e52548377d5a5 WatchSource:0}: Error finding container 9cb86091e449c481ac4a685564801b24f16b74ad77606d2f4f2e52548377d5a5: Status 404 returned error can't find the container with id 9cb86091e449c481ac4a685564801b24f16b74ad77606d2f4f2e52548377d5a5 Jul 14 20:34:51.314511 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:51.314421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" event={"ID":"14f2a50e-462f-4727-9c99-f688ecd1bc30","Type":"ContainerStarted","Data":"b0a7b8f3d7101ba8d91c46a97b65469ee6cbd02cfe347301b7c4966ebd2ce952"} Jul 14 20:34:51.314511 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:51.314506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" event={"ID":"14f2a50e-462f-4727-9c99-f688ecd1bc30","Type":"ContainerStarted","Data":"9cb86091e449c481ac4a685564801b24f16b74ad77606d2f4f2e52548377d5a5"} Jul 14 20:34:51.314781 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:51.314671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:51.333266 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:51.333219 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" podStartSLOduration=1.333204073 podStartE2EDuration="1.333204073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:34:51.331030859 +0000 UTC m=+2841.175367734" watchObservedRunningTime="2026-07-14 20:34:51.333204073 +0000 UTC m=+2841.177540947" Jul 14 20:34:52.990983 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:52.990944 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bczlc_7c0a352d-44e2-423c-b0af-758ed4742235/dns/0.log" Jul 14 20:34:53.014145 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:53.014120 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bczlc_7c0a352d-44e2-423c-b0af-758ed4742235/kube-rbac-proxy/0.log" Jul 14 20:34:53.170100 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:53.170062 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dltdc_4e6499e8-d01c-4ff3-b126-6ee1237e17c4/dns-node-resolver/0.log" Jul 14 20:34:53.762206 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:53.762175 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-849757d945-bdmfj_5155ceaf-504a-4a44-9fda-ab319c15539a/registry/0.log" Jul 14 20:34:53.883289 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:53.883258 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vw69b_ecffb1fb-7b0d-4884-951f-24ce4fc3ae1d/node-ca/0.log" Jul 14 20:34:54.851086 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:54.851049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67884mdb_250b0041-6ff0-4dff-b146-b2e02e122934/istio-proxy/0.log" Jul 14 20:34:55.092754 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:55.092717 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/0.log" Jul 14 20:34:55.156649 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:55.156566 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/1.log" Jul 14 20:34:56.002298 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:56.002257 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8vpfn_edcb30b6-4adb-4e40-84e2-1ac353237d17/serve-healthcheck-canary/0.log" Jul 14 20:34:56.689809 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:56.689780 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdlcb_1ef62b39-b6dc-4165-80ef-e7aa6163f239/kube-rbac-proxy/0.log" Jul 14 20:34:56.715987 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:56.715960 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdlcb_1ef62b39-b6dc-4165-80ef-e7aa6163f239/exporter/0.log" Jul 14 20:34:56.740361 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:56.740320 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdlcb_1ef62b39-b6dc-4165-80ef-e7aa6163f239/extractor/0.log" Jul 14 20:34:57.336195 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:57.336153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tnh4v/perf-node-gather-daemonset-pwlk4" Jul 14 20:34:59.459772 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:59.459733 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-5b6jd_07ae2cb9-a573-41ef-b100-3f643925c6c4/manager/0.log" Jul 14 20:34:59.631975 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:59.631945 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-gk7lj_0894dd72-2728-437f-a9de-eb6d75738a92/manager/0.log" Jul 14 20:34:59.657978 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:34:59.657949 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c597f769-gk7lj_0894dd72-2728-437f-a9de-eb6d75738a92/copy-manifests/0.log" Jul 14 20:35:00.572529 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:00.572389 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67884mdb_250b0041-6ff0-4dff-b146-b2e02e122934/istio-proxy/0.log" Jul 14 20:35:00.742440 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:00.742405 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/0.log" Jul 14 20:35:00.808387 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:00.808352 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/1.log" Jul 14 20:35:01.100440 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:01.100415 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77d6d59f96-62vfx_44a10a91-5fe6-4150-9567-874fdf632116/manager/0.log" Jul 14 20:35:01.168846 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:01.168809 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wbjc2_cdab5eea-37f6-4969-8919-89db79a70d03/openshift-lws-operator/0.log" Jul 14 20:35:02.132050 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:02.132017 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb67884mdb_250b0041-6ff0-4dff-b146-b2e02e122934/istio-proxy/0.log" Jul 14 20:35:02.298176 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:02.298129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/0.log" Jul 14 20:35:02.376658 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:02.376331 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-v6dq6_6e9ecf78-8596-49c1-90d9-e0c8498c7494/istio-proxy/1.log" Jul 14 20:35:07.605265 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:07.605195 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6ttml_19dc80ba-52eb-4761-918f-e5b4be8ee2df/kube-multus/0.log" Jul 14 20:35:08.005958 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:08.005905 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tpn29_80dfc55c-0e33-4820-b465-87fa724e7757/kube-multus-additional-cni-plugins/0.log" Jul 14 20:35:08.028564 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:08.028527 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tpn29_80dfc55c-0e33-4820-b465-87fa724e7757/egress-router-binary-copy/0.log" Jul 14 20:35:08.060270 ip-10-0-129-28 kubenswrapper[2574]: I0714 20:35:08.060238 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tpn29_80dfc55c-0e33-4820-b465-87fa724e7757/cni-plugins/0.log"