Jul 05 19:29:59.408352 ip-10-0-143-177 systemd[1]: Starting Kubernetes Kubelet... Jul 05 19:29:59.858428 ip-10-0-143-177 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:59.858428 ip-10-0-143-177 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 05 19:29:59.858428 ip-10-0-143-177 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:59.858428 ip-10-0-143-177 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 05 19:29:59.858428 ip-10-0-143-177 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:59.859537 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.859451 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869008 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869032 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869036 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869039 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869042 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.869035 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869044 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869048 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869065 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869068 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869072 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869074 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869078 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869080 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869083 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869085 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869088 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869090 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869093 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869096 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869098 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869101 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869103 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869105 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869108 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869110 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.869285 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869113 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869124 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869127 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869129 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869131 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869134 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869137 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869140 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869143 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869145 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869147 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869150 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869152 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869155 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869158 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869160 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869163 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869165 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869167 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869170 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.869756 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869172 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869175 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869177 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869179 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869182 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869184 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869187 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869189 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869192 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869194 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869197 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869199 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869202 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869204 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869206 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869209 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869213 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869218 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869222 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869225 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.870290 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869227 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869230 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869232 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869235 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869238 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869240 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869243 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869246 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869248 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869251 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869255 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869259 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869261 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869266 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869270 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869272 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869276 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869279 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869287 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.870773 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869290 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869294 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869297 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869300 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869303 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869306 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869308 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869311 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869313 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869316 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869318 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869321 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869325 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869329 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.869332 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870134 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870141 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870143 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870147 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870149 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.871245 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870152 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870155 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870159 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870163 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870166 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870169 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870172 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870176 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870179 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870182 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870185 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870187 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870196 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870199 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870202 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870204 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870207 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870209 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870212 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.871727 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870214 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870217 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870219 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870222 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870226 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870228 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870231 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870233 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870236 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870239 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870242 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870245 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870247 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870250 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870252 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870255 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870257 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870259 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870262 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870264 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.872218 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870266 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870269 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870271 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870273 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870276 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870278 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870286 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870289 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870291 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870294 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870297 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870299 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870302 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870304 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870307 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870310 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870313 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870317 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870320 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.872717 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870323 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870326 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870329 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870332 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870334 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870337 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870339 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870342 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870344 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870346 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870349 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870351 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870354 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870357 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870361 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870364 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870366 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870369 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870372 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870374 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.873205 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870382 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870384 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870387 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870389 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870391 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870394 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870396 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870399 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870401 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870404 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870406 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870409 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870411 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870414 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870416 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870418 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870900 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870906 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870909 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870912 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.873685 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870915 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870917 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870920 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870923 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870925 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870927 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870930 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870933 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870935 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870938 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870940 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870942 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870946 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870949 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870951 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870954 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870956 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870959 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870962 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.874207 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870964 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870967 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870969 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870972 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870974 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870977 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870979 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870982 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870984 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870988 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870991 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870993 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870995 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.870998 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871000 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871003 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871005 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871008 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871011 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871013 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.874660 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871016 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871018 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871021 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871023 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871026 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871029 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871031 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871034 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871036 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871039 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871041 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871044 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871046 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871063 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871066 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871069 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871075 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871078 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871081 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871085 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.875166 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871087 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871091 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871094 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871097 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871101 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871105 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871108 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871111 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871113 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871115 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871118 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871120 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871123 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871125 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871128 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871131 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871133 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871136 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871138 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.875676 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871141 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871144 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871146 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871149 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871151 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871156 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871158 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871161 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871163 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871166 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871168 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871170 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871173 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871175 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871178 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871180 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871183 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871253 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871261 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871268 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871272 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 05 19:29:59.876137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871276 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871279 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871283 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871290 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871293 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871296 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871299 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871302 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871305 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871308 2581 flags.go:64] FLAG: --cgroup-root="" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871311 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871313 2581 flags.go:64] FLAG: --client-ca-file="" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871317 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871320 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871324 2581 flags.go:64] FLAG: --cluster-domain="" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871326 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871330 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871333 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871337 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871340 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871343 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871346 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871349 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871352 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 05 19:29:59.876668 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871355 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871358 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871361 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871364 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871368 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871371 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871374 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871376 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871381 2581 flags.go:64] FLAG: --enable-server="true" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871384 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871389 2581 flags.go:64] FLAG: --event-burst="100" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871392 2581 flags.go:64] FLAG: --event-qps="50" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871394 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871397 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871400 2581 flags.go:64] FLAG: --eviction-hard="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871404 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871406 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871409 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871412 2581 flags.go:64] FLAG: --eviction-soft="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871415 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871417 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871421 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871423 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871426 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871430 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 05 19:29:59.877257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871433 2581 flags.go:64] FLAG: --feature-gates="" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871436 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871440 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871442 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871446 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871448 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871451 2581 flags.go:64] FLAG: --help="false" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871454 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-177.ec2.internal" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871457 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871460 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871462 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871465 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871469 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871472 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871474 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871479 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871481 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871484 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871487 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871490 2581 flags.go:64] FLAG: --kube-reserved="" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871492 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871495 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871498 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871501 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 05 19:29:59.877959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871503 2581 flags.go:64] FLAG: --lock-file="" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871506 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871508 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871511 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871516 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871520 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871522 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871527 2581 flags.go:64] FLAG: --logging-format="text" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871530 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871534 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871536 2581 flags.go:64] FLAG: --manifest-url="" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871539 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871543 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871546 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871550 2581 flags.go:64] FLAG: --max-pods="110" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871553 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871556 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871558 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871561 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871564 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871566 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871569 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871576 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871580 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871583 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 05 19:29:59.878602 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871586 2581 flags.go:64] FLAG: --pod-cidr="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871589 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871594 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871597 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871600 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871603 2581 flags.go:64] FLAG: --port="10250" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871605 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871608 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085bb7cbec930241b" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871611 2581 flags.go:64] FLAG: --qos-reserved="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871614 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871617 2581 flags.go:64] FLAG: --register-node="true" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871620 2581 flags.go:64] FLAG: --register-with-taints="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871624 2581 flags.go:64] FLAG: --registry-burst="10" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871627 2581 flags.go:64] FLAG: --registry-qps="5" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871630 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871635 2581 flags.go:64] FLAG: --reserved-memory="" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871638 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871641 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871644 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871647 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871649 2581 flags.go:64] FLAG: --runonce="false" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871652 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871655 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871658 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871660 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 05 19:29:59.879250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871663 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871666 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871669 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871671 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871674 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871678 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871681 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871683 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871686 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871689 2581 flags.go:64] FLAG: --system-cgroups="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871692 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871697 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871700 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871702 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871706 2581 flags.go:64] FLAG: --tls-min-version="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871709 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871711 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871714 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871717 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871719 2581 flags.go:64] FLAG: --v="2" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871723 2581 flags.go:64] FLAG: --version="false" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871727 2581 flags.go:64] FLAG: --vmodule="" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871732 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.871736 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871856 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.879852 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871860 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871863 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871865 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871868 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871870 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871873 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871875 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871878 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871880 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871883 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871885 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871888 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871893 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871895 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871898 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871900 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871902 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871905 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871907 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871910 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.880515 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871912 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871914 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871917 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871919 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871921 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871924 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871926 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871929 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871931 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871936 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871938 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871941 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871943 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871946 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871948 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871951 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871953 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871956 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871958 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871961 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.881004 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871963 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871966 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871968 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871971 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871975 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871977 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871980 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871982 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871985 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871987 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871989 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871992 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871994 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.871999 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872002 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872005 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872008 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872011 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872014 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.881499 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872016 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872019 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872022 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872026 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872028 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872031 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872033 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872035 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872038 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872040 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872043 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872045 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872047 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872064 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872067 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872070 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872073 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872077 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872079 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872082 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.881956 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872084 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872089 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872092 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872096 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872100 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872103 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872106 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872108 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872110 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872113 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872115 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872118 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872120 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872122 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872126 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872129 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872131 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872134 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.882453 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.872136 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.872766 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.879939 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.879957 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.879977 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.879983 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880038 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880043 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880046 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880067 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880072 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880075 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880077 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880080 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880083 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.882900 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880085 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880088 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880090 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880093 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880095 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880098 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880100 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880102 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880105 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880107 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880110 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880112 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880115 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880117 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880119 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880122 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880124 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880126 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880129 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.883304 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880132 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880136 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880139 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880141 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880144 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880146 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880149 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880152 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880154 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880157 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880159 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880162 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880164 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880166 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880169 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880171 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880174 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880176 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880179 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880181 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.883783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880184 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880186 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880189 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880191 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880193 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880196 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880198 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880201 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880203 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880207 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880211 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880214 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880216 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880220 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880224 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880226 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880229 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880232 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880234 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880236 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.884258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880239 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880241 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880244 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880246 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880249 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880252 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880254 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880256 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880259 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880261 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880264 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880266 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880269 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880271 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880274 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880277 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880279 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880282 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880284 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880287 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.884752 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880289 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880291 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880293 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880297 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880301 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880304 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880307 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880310 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880313 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880315 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880319 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.880324 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880457 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880461 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:59.885264 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880464 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880467 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880470 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880472 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880475 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880477 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880480 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880483 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880485 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880488 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880490 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880493 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880496 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880498 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880501 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880503 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880506 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880508 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880510 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:59.885617 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880513 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880515 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880518 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880520 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880523 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880527 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880529 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880532 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880535 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880537 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880540 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880543 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880547 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880550 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880552 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880555 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880557 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880559 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880563 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:59.886082 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880566 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880569 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880571 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880574 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880576 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880578 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880581 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880583 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880585 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880588 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880591 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880593 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880596 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880598 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880600 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880603 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880605 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880607 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880610 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880613 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:59.886547 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880616 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880619 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880621 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880624 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880626 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880628 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880630 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880633 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880635 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880637 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880640 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880642 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880644 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880647 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880649 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880651 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880654 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880656 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880658 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880661 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:59.887037 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880663 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880666 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880668 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880670 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880672 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880675 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880677 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880679 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880682 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880684 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880688 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880691 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880694 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880697 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880700 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880702 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880705 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880708 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:59.887541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:29:59.880710 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.880715 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.880824 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.883084 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.884693 2581 server.go:1019] "Starting client certificate rotation" Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.884793 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.884822 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 19:29:59.887974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.887504 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 19:29:59.888805 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.888792 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 19:29:59.902527 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.902483 2581 log.go:25] "Validated CRI v1 runtime API" Jul 05 19:29:59.906743 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.906720 2581 log.go:25] "Validated CRI v1 image API" Jul 05 19:29:59.907618 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.907602 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 05 19:29:59.910105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.910080 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 19:29:59.911131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.911108 2581 fs.go:135] Filesystem UUIDs: map[4822b97f-c294-4a69-af2e-ef204a116527:/dev/nvme0n1p3 5c4c1850-0ed5-481f-9504-c6392e46ea6f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 05 19:29:59.911199 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.911127 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 05 19:29:59.916312 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.916195 2581 manager.go:216] Machine: {Timestamp:2026-07-05 19:29:59.914843017 +0000 UTC m=+0.370626438 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100306 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b0a6dcf8ab8fcabf2592d4c5ec330 SystemUUID:ec2b0a6d-cf8a-b8fc-abf2-592d4c5ec330 BootID:57dcab21-74f1-4313-857a-87c75661a22a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:21:40:89:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:21:40:89:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:6c:74:b7:f0:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 05 19:29:59.916312 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.916306 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 05 19:29:59.916434 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.916391 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 05 19:29:59.920522 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.920497 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 05 19:29:59.920669 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.920524 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 05 19:29:59.922086 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.922074 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 05 19:29:59.922120 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.922088 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 05 19:29:59.922120 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.922095 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 19:29:59.922120 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.922114 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 19:29:59.923197 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.923186 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 05 19:29:59.927640 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927628 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 05 19:29:59.927781 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927770 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 05 19:29:59.927826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927816 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 05 19:29:59.927855 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927841 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 05 19:29:59.927885 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927863 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 05 19:29:59.927885 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927876 2581 kubelet.go:398] "Adding apiserver pod source" Jul 05 19:29:59.927955 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927892 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 05 19:29:59.927955 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.927939 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fz695" Jul 05 19:29:59.928280 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.928268 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 19:29:59.928327 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.928287 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 19:29:59.932497 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.932474 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 05 19:29:59.934344 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.934331 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 05 19:29:59.934410 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.934361 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 05 19:29:59.936071 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936040 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fz695" Jul 05 19:29:59.936347 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936334 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936352 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936358 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936364 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936371 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936377 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936384 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 05 19:29:59.936406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936406 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 05 19:29:59.936591 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936416 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 05 19:29:59.936591 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936422 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 05 19:29:59.936591 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936437 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 05 19:29:59.936591 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.936443 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 05 19:29:59.937289 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.937277 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 05 19:29:59.937289 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.937289 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 05 19:29:59.939100 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:29:59.939080 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 05 19:29:59.939156 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:29:59.939124 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 05 19:29:59.940756 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.940743 2581 server.go:1268] "Started kubelet" Jul 05 19:29:59.940887 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.940841 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 05 19:29:59.941560 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.941490 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 05 19:29:59.941660 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.941587 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 05 19:29:59.941610 ip-10-0-143-177 systemd[1]: Started Kubernetes Kubelet. Jul 05 19:29:59.943191 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.943172 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 05 19:29:59.943434 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.943423 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 05 19:29:59.947421 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:29:59.947399 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 05 19:29:59.952385 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.952369 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 05 19:29:59.952385 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.952382 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 05 19:29:59.952850 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:29:59.952475 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:29:59.952850 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.952824 2581 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-177.ec2.internal" not found Jul 05 19:29:59.953120 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953102 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 05 19:29:59.953227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953132 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 05 19:29:59.953227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953204 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 05 19:29:59.953366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953341 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 05 19:29:59.953366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953357 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 05 19:29:59.953513 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953490 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 05 19:29:59.953513 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953512 2581 factory.go:55] Registering systemd factory Jul 05 19:29:59.953634 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953521 2581 factory.go:223] Registration of the systemd container factory successfully Jul 05 19:29:59.953874 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953744 2581 factory.go:153] Registering CRI-O factory Jul 05 19:29:59.953927 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953877 2581 factory.go:223] Registration of the crio container factory successfully Jul 05 19:29:59.953927 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953901 2581 factory.go:103] Registering Raw factory Jul 05 19:29:59.953927 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.953915 2581 manager.go:1196] Started watching for new ooms in manager Jul 05 19:29:59.954108 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.954093 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:29:59.955089 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.955073 2581 manager.go:349] Starting recovery of all containers Jul 05 19:29:59.956013 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:29:59.955991 2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-177.ec2.internal\" not found" node="ip-10-0-143-177.ec2.internal" Jul 05 19:29:59.965213 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.964922 2581 manager.go:354] Recovery completed Jul 05 19:29:59.967146 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.967131 2581 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-177.ec2.internal" not found Jul 05 19:29:59.969478 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.969466 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:59.973927 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.973910 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:59.974016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.973938 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:59.974016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.973949 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:59.974498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.974485 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 05 19:29:59.974498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.974498 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 05 19:29:59.974612 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.974516 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 05 19:29:59.974612 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.974487 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:59.975271 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.975255 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:59.975325 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.975291 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:59.975325 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.975307 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:59.976780 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.976767 2581 policy_none.go:49] "None policy: Start" Jul 05 19:29:59.976851 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.976799 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 05 19:29:59.976851 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.976814 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 05 19:29:59.978781 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:29:59.978754 2581 policy_none.go:47] "Start" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.021696 2581 manager.go:337] "Starting Device Plugin manager" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.021780 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.021790 2581 server.go:87] "Starting device plugin registration server" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.022071 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.022084 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.022166 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.022241 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.022251 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.023153 2581 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-143-177.ec2.internal" not found Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.023204 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 05 19:30:00.034252 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.023270 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.052837 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.052806 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 05 19:30:00.054167 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.054143 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 05 19:30:00.054273 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.054196 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 05 19:30:00.054273 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.054220 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 05 19:30:00.054273 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.054255 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 05 19:30:00.056199 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.056183 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:00.122317 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.122225 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:30:00.123306 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.123284 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:30:00.123444 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.123320 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:30:00.123444 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.123336 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:30:00.123444 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.123366 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.130955 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.130938 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.131031 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.130962 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-177.ec2.internal\": node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.143097 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.143046 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.155040 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.155008 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal"] Jul 05 19:30:00.155138 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.155107 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:30:00.156661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.156643 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:30:00.156734 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.156677 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:30:00.156734 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.156691 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:30:00.158931 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.158916 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:30:00.159077 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159041 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.159145 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159103 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:30:00.159753 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159738 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:30:00.159826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159760 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:30:00.159826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159780 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:30:00.159826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159791 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:30:00.159826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159764 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:30:00.159964 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.159841 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:30:00.161949 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.161931 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.162012 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.161964 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:30:00.162803 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.162785 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:30:00.162894 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.162822 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:30:00.162894 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.162839 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:30:00.175739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.175715 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-177.ec2.internal\" not found" node="ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.179349 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.179329 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-177.ec2.internal\" not found" node="ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.243381 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.243345 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.255544 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.255508 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.255671 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.255551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.255671 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.255578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd4ad6f0c2d4023e232d083fe9e47834-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"bd4ad6f0c2d4023e232d083fe9e47834\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.344363 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.344311 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.356704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.356757 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.356757 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd4ad6f0c2d4023e232d083fe9e47834-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"bd4ad6f0c2d4023e232d083fe9e47834\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.356828 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd4ad6f0c2d4023e232d083fe9e47834-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"bd4ad6f0c2d4023e232d083fe9e47834\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.356864 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356818 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.356864 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.356793 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9abece3269131efc7d2a5033be560fe2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"9abece3269131efc7d2a5033be560fe2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.445197 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.445123 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.477597 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.477571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.481515 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.481496 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:00.546291 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.546255 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.646706 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.646674 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.747286 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.747224 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.847715 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.847688 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:00.884224 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.884198 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 05 19:30:00.884879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.884335 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 05 19:30:00.884879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.884368 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 05 19:30:00.942728 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.942676 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-04 19:25:00 +0000 UTC" deadline="2028-02-11 10:13:28.926880322 +0000 UTC" Jul 05 19:30:00.942728 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:00.942726 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14054h43m27.98415846s" Jul 05 19:30:00.947790 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:00.947769 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:01.048491 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:01.048419 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:01.089362 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:01.089330 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abece3269131efc7d2a5033be560fe2.slice/crio-1bef70c2423b756efce254b8104b9dee5a8f0f34a71b9f03fde29560dac0ec9a WatchSource:0}: Error finding container 1bef70c2423b756efce254b8104b9dee5a8f0f34a71b9f03fde29560dac0ec9a: Status 404 returned error can't find the container with id 1bef70c2423b756efce254b8104b9dee5a8f0f34a71b9f03fde29560dac0ec9a Jul 05 19:30:01.089958 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:01.089930 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4ad6f0c2d4023e232d083fe9e47834.slice/crio-a6cb9525932580a949a9fd91bacad220632ad1822f9b89e855e7ad56d73f7bde WatchSource:0}: Error finding container a6cb9525932580a949a9fd91bacad220632ad1822f9b89e855e7ad56d73f7bde: Status 404 returned error can't find the container with id a6cb9525932580a949a9fd91bacad220632ad1822f9b89e855e7ad56d73f7bde Jul 05 19:30:01.095163 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.095143 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:30:01.149366 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:01.149321 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Jul 05 19:30:01.191332 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.191308 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:01.253599 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.253577 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Jul 05 19:30:01.264169 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.264151 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 19:30:01.266101 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.266089 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Jul 05 19:30:01.274445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.274431 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 19:30:01.434070 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.434044 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:01.923069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.922959 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:01.929345 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.929317 2581 apiserver.go:52] "Watching apiserver" Jul 05 19:30:01.937239 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.937213 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 05 19:30:01.938585 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.938556 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-tqkck","openshift-ovn-kubernetes/ovnkube-node-5qtff","kube-system/global-pull-secret-syncer-6srrh","kube-system/konnectivity-agent-vhv6b","openshift-cluster-node-tuning-operator/tuned-26dbz","openshift-dns/node-resolver-jdm75","openshift-image-registry/node-ca-jkxjb","openshift-multus/multus-756dc","openshift-network-diagnostics/network-check-target-72jl7","openshift-network-operator/iptables-alerter-4jzkw","kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal","openshift-multus/multus-additional-cni-plugins-6wzpw"] Jul 05 19:30:01.941320 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.941297 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:01.943601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.943581 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-756dc" Jul 05 19:30:01.944666 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.944498 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.944666 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.944528 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.944666 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.944544 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 05 19:30:01.944666 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.944564 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q8qvg\"" Jul 05 19:30:01.945911 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.945887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:01.946013 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:01.945969 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:01.946197 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.946178 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 05 19:30:01.946285 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.946236 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 05 19:30:01.946581 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.946555 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.947260 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.947234 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.947381 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.947364 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jg89k\"" Jul 05 19:30:01.951111 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.951092 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:01.952998 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.952978 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 05 19:30:01.953658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.953640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.955514 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.955493 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 05 19:30:01.955604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.955528 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-92clt\"" Jul 05 19:30:01.955804 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.955725 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 05 19:30:01.955863 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.955851 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:01.956545 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.956523 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.956656 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.956570 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnlqg\"" Jul 05 19:30:01.956758 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.956739 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.958427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.958405 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:01.958521 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:01.958471 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:01.959016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.958992 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5lcm9\"" Jul 05 19:30:01.959124 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.959003 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.959191 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.959161 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.960938 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.960920 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:01.961028 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:01.960970 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:01.964001 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.963982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:01.965001 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.964978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-hostroot\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965096 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-systemd\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965096 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965068 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-tuned\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965124 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-conf-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49904b6-fdf6-47d2-8235-bd1927aff2f9-konnectivity-ca\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-lib-modules\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-daemon-config\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965230 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-etc-kubernetes\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49904b6-fdf6-47d2-8235-bd1927aff2f9-agent-certs\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:01.965316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d4538-2d34-496f-8aec-a84417bda656-tmp-dir\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-netns\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965344 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-modprobe-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-sys\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjh2\" (UniqueName: \"kubernetes.io/projected/dd679826-cb58-4de2-83b2-378bd372dbd0-kube-api-access-2rjh2\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a8d4538-2d34-496f-8aec-a84417bda656-hosts-file\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wsd\" (UniqueName: \"kubernetes.io/projected/5a8d4538-2d34-496f-8aec-a84417bda656-kube-api-access-n6wsd\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965506 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c52aea1b-58e4-41ac-ac04-db4a90d9c794-serviceca\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-multus\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-system-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965589 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-os-release\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.965655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cni-binary-copy\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-k8s-cni-cncf-io\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-var-lib-kubelet\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-socket-dir-parent\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysconfig\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-kubernetes\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-host\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.965978 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-tmp\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c52aea1b-58e4-41ac-ac04-db4a90d9c794-host\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966084 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cnibin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8z8r\" (UniqueName: \"kubernetes.io/projected/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-kube-api-access-t8z8r\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-conf\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966184 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9g6d\" (UniqueName: \"kubernetes.io/projected/c52aea1b-58e4-41ac-ac04-db4a90d9c794-kube-api-access-w9g6d\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-kubelet\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-multus-certs\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966328 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-kubelet-config\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-dbus\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966405 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966429 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-run\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966476 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-bin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:01.966794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.966490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:01.967265 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967244 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.967388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967372 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 05 19:30:01.967518 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967256 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 05 19:30:01.967597 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967584 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 05 19:30:01.967737 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967716 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v7q4h\"" Jul 05 19:30:01.967841 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967753 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 05 19:30:01.967841 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.967755 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.970691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.969190 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.970691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.969363 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:01.970691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.969460 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 05 19:30:01.970691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.969603 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.970691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.969363 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nw77c\"" Jul 05 19:30:01.971786 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.971765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:01.971933 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.971902 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 05 19:30:01.972406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.972378 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjjsq\"" Jul 05 19:30:01.972513 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.972437 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 05 19:30:01.972727 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.972706 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 19:30:01.974502 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.974474 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.974750 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.974728 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 05 19:30:01.975471 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.975256 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9z7w\"" Jul 05 19:30:01.975471 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.975437 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.998934 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:01.998910 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gvp9d" Jul 05 19:30:02.007796 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.007775 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gvp9d" Jul 05 19:30:02.054613 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.054580 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 05 19:30:02.059185 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.059142 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"9abece3269131efc7d2a5033be560fe2","Type":"ContainerStarted","Data":"1bef70c2423b756efce254b8104b9dee5a8f0f34a71b9f03fde29560dac0ec9a"} Jul 05 19:30:02.060370 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.060351 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" event={"ID":"bd4ad6f0c2d4023e232d083fe9e47834","Type":"ContainerStarted","Data":"a6cb9525932580a949a9fd91bacad220632ad1822f9b89e855e7ad56d73f7bde"} Jul 05 19:30:02.067590 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-systemd\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067598 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-tuned\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-host-slash\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.067690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067645 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-system-cni-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.067690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-var-lib-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-etc-kubernetes\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-systemd\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-netns\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067791 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-netns\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-modprobe-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-sys\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-sys\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.067907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067906 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-etc-kubernetes\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjh2\" (UniqueName: \"kubernetes.io/projected/dd679826-cb58-4de2-83b2-378bd372dbd0-kube-api-access-2rjh2\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-kubelet\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.067980 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068005 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-modprobe-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-multus\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-system-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068174 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-multus\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-os-release\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068219 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-system-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-k8s-cni-cncf-io\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068251 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-cni-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.068266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068263 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-tmp\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068280 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-k8s-cni-cncf-io\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-socket-dir-parent\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-os-release\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-host\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-systemd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-node-log\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068376 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-socket-dir-parent\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068382 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-sys-fs\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068393 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-host\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8z8r\" (UniqueName: \"kubernetes.io/projected/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-kube-api-access-t8z8r\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068436 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9g6d\" (UniqueName: \"kubernetes.io/projected/c52aea1b-58e4-41ac-ac04-db4a90d9c794-kube-api-access-w9g6d\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068457 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-systemd-units\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068476 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjmv\" (UniqueName: \"kubernetes.io/projected/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-kube-api-access-tbjmv\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzvw\" (UniqueName: \"kubernetes.io/projected/45c5b67a-a46d-45c7-a3a1-03de771be4fc-kube-api-access-5mzvw\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.069034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-kubelet\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-kubelet-config\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-run\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-netd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068722 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-kubelet-config\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068733 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-kubelet\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068764 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-run\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068788 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-config\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068819 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dd4\" (UniqueName: \"kubernetes.io/projected/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-kube-api-access-86dd4\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068864 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cc6w\" (UniqueName: \"kubernetes.io/projected/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kube-api-access-5cc6w\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-hostroot\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068927 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-log-socket\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068971 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-hostroot\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.068999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-socket-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.069816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-conf-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49904b6-fdf6-47d2-8235-bd1927aff2f9-konnectivity-ca\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069096 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-conf-dir\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069132 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-lib-modules\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-os-release\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-lib-modules\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-daemon-config\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49904b6-fdf6-47d2-8235-bd1927aff2f9-agent-certs\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d4538-2d34-496f-8aec-a84417bda656-tmp-dir\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a8d4538-2d34-496f-8aec-a84417bda656-hosts-file\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069468 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wsd\" (UniqueName: \"kubernetes.io/projected/5a8d4538-2d34-496f-8aec-a84417bda656-kube-api-access-n6wsd\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c52aea1b-58e4-41ac-ac04-db4a90d9c794-serviceca\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b26ef5e5-d807-44d7-9d12-905c17073d16-ovn-node-metrics-cert\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-script-lib\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cni-binary-copy\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069592 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-var-lib-kubelet\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069605 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5a8d4538-2d34-496f-8aec-a84417bda656-hosts-file\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069619 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-ovn\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.070688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b49904b6-fdf6-47d2-8235-bd1927aff2f9-konnectivity-ca\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-bin\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-env-overrides\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069732 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ntr\" (UniqueName: \"kubernetes.io/projected/b26ef5e5-d807-44d7-9d12-905c17073d16-kube-api-access-42ntr\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069786 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-device-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysconfig\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069816 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-multus-daemon-config\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-kubernetes\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-kubernetes\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c52aea1b-58e4-41ac-ac04-db4a90d9c794-host\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysconfig\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.069955 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-slash\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070008 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-var-lib-kubelet\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070028 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c52aea1b-58e4-41ac-ac04-db4a90d9c794-host\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070041 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-iptables-alerter-script\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.071253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070071 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cnibin\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070097 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c52aea1b-58e4-41ac-ac04-db4a90d9c794-serviceca\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-d\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070175 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cnibin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-conf\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-netns\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070263 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cnibin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070340 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-sysctl-conf\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-etc-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070393 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-registration-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070437 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-cni-binary-copy\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-multus-certs\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-dbus\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070484 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-run-multus-certs\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070496 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070527 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-bin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.071862 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.070569 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070589 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3003f33-196f-47f9-ad7b-55f6f99c48eb-dbus\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-host-var-lib-cni-bin\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.070614 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a8d4538-2d34-496f-8aec-a84417bda656-tmp-dir\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.070634 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.570618179 +0000 UTC m=+3.026401589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.072122 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-tmp\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.072291 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd679826-cb58-4de2-83b2-378bd372dbd0-etc-tuned\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.072664 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.072453 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b49904b6-fdf6-47d2-8235-bd1927aff2f9-agent-certs\") pod \"konnectivity-agent-vhv6b\" (UID: \"b49904b6-fdf6-47d2-8235-bd1927aff2f9\") " pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:02.078792 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.078763 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8z8r\" (UniqueName: \"kubernetes.io/projected/94267a0b-11cf-4f85-8c61-6c7b4bd088ab-kube-api-access-t8z8r\") pod \"multus-756dc\" (UID: \"94267a0b-11cf-4f85-8c61-6c7b4bd088ab\") " pod="openshift-multus/multus-756dc" Jul 05 19:30:02.078881 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.078799 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wsd\" (UniqueName: \"kubernetes.io/projected/5a8d4538-2d34-496f-8aec-a84417bda656-kube-api-access-n6wsd\") pod \"node-resolver-jdm75\" (UID: \"5a8d4538-2d34-496f-8aec-a84417bda656\") " pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.079218 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.079198 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9g6d\" (UniqueName: \"kubernetes.io/projected/c52aea1b-58e4-41ac-ac04-db4a90d9c794-kube-api-access-w9g6d\") pod \"node-ca-jkxjb\" (UID: \"c52aea1b-58e4-41ac-ac04-db4a90d9c794\") " pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.079314 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.079257 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjh2\" (UniqueName: \"kubernetes.io/projected/dd679826-cb58-4de2-83b2-378bd372dbd0-kube-api-access-2rjh2\") pod \"tuned-26dbz\" (UID: \"dd679826-cb58-4de2-83b2-378bd372dbd0\") " pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.171397 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171357 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjmv\" (UniqueName: \"kubernetes.io/projected/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-kube-api-access-tbjmv\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.171566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzvw\" (UniqueName: \"kubernetes.io/projected/45c5b67a-a46d-45c7-a3a1-03de771be4fc-kube-api-access-5mzvw\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.171566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171433 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-netd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.171566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171497 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-netd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.171566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-config\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.171566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dd4\" (UniqueName: \"kubernetes.io/projected/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-kube-api-access-86dd4\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171600 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cc6w\" (UniqueName: \"kubernetes.io/projected/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kube-api-access-5cc6w\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-log-socket\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.171722 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.171794 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.671774131 +0000 UTC m=+3.127557545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.171826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-log-socket\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-socket-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-os-release\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171904 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b26ef5e5-d807-44d7-9d12-905c17073d16-ovn-node-metrics-cert\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171956 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-script-lib\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171968 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-os-release\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171974 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-socket-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.171984 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-ovn\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172031 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-ovn\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-bin\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-env-overrides\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172115 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-cni-bin\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172214 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ntr\" (UniqueName: \"kubernetes.io/projected/b26ef5e5-d807-44d7-9d12-905c17073d16-kube-api-access-42ntr\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-device-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-slash\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172377 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-iptables-alerter-script\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172400 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cnibin\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172424 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172453 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-netns\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172478 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-config\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172477 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-etc-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-slash\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172584 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-registration-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-host-slash\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-system-cni-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-registration-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-var-lib-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.172948 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172769 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-ovnkube-script-lib\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172787 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cnibin\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-netns\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-kubelet\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172822 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-host-slash\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172889 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45c5b67a-a46d-45c7-a3a1-03de771be4fc-system-cni-dir\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.172914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173043 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-kubelet\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173118 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173228 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-device-dir\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173308 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b26ef5e5-d807-44d7-9d12-905c17073d16-env-overrides\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173411 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-etc-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173461 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173489 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173506 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-var-lib-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173574 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-run-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.173848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173599 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-openvswitch\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173638 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-etc-selinux\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-systemd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-node-log\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173719 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-iptables-alerter-script\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173736 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-sys-fs\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173761 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-run-systemd\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-systemd-units\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173791 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-sys-fs\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173809 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-node-log\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173839 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-systemd-units\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.173888 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b26ef5e5-d807-44d7-9d12-905c17073d16-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.174601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.174088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45c5b67a-a46d-45c7-a3a1-03de771be4fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.175241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.175174 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b26ef5e5-d807-44d7-9d12-905c17073d16-ovn-node-metrics-cert\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.181768 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.181742 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:02.181886 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.181778 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:02.181886 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.181794 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:02.181886 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.181850 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.68183401 +0000 UTC m=+3.137617420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:02.184256 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.184231 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjmv\" (UniqueName: \"kubernetes.io/projected/5c654201-8758-4ce8-9b5c-0f6d47f62e8f-kube-api-access-tbjmv\") pod \"iptables-alerter-4jzkw\" (UID: \"5c654201-8758-4ce8-9b5c-0f6d47f62e8f\") " pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.184534 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.184505 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzvw\" (UniqueName: \"kubernetes.io/projected/45c5b67a-a46d-45c7-a3a1-03de771be4fc-kube-api-access-5mzvw\") pod \"multus-additional-cni-plugins-6wzpw\" (UID: \"45c5b67a-a46d-45c7-a3a1-03de771be4fc\") " pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.184598 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.184540 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ntr\" (UniqueName: \"kubernetes.io/projected/b26ef5e5-d807-44d7-9d12-905c17073d16-kube-api-access-42ntr\") pod \"ovnkube-node-5qtff\" (UID: \"b26ef5e5-d807-44d7-9d12-905c17073d16\") " pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.185245 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.185216 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cc6w\" (UniqueName: \"kubernetes.io/projected/1049dfed-1d3f-499e-9c9b-ac4175a7fd12-kube-api-access-5cc6w\") pod \"aws-ebs-csi-driver-node-v9flv\" (UID: \"1049dfed-1d3f-499e-9c9b-ac4175a7fd12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.185696 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.185509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dd4\" (UniqueName: \"kubernetes.io/projected/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-kube-api-access-86dd4\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.260047 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.260006 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jkxjb" Jul 05 19:30:02.268286 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.268256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-756dc" Jul 05 19:30:02.279467 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.279441 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:02.284657 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.284620 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26dbz" Jul 05 19:30:02.288353 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.288331 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:02.292753 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.292726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jdm75" Jul 05 19:30:02.299690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.299670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:02.311355 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.311328 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4jzkw" Jul 05 19:30:02.319175 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.319151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" Jul 05 19:30:02.325726 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.325705 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" Jul 05 19:30:02.577554 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.577475 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:02.577727 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.577601 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.577727 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.577678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.577654186 +0000 UTC m=+4.033437638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.678333 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.678303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:02.678477 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.678416 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.678521 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.678477 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.678457575 +0000 UTC m=+4.134240985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.778646 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:02.778606 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:02.778819 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.778733 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:02.778819 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.778758 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:02.778819 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.778773 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:02.779012 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:02.778834 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.778819249 +0000 UTC m=+4.234602813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:03.011446 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.011408 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 19:25:02 +0000 UTC" deadline="2028-03-12 19:06:14.160172654 +0000 UTC" Jul 05 19:30:03.011446 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.011441 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14783h36m11.14873621s" Jul 05 19:30:03.054760 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.054635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:03.054847 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.054750 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:03.078258 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.078228 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52aea1b_58e4_41ac_ac04_db4a90d9c794.slice/crio-d158d1e519bbb2ff90c3d646a9742f85fcef0405be4b0366062739bf1356bf10 WatchSource:0}: Error finding container d158d1e519bbb2ff90c3d646a9742f85fcef0405be4b0366062739bf1356bf10: Status 404 returned error can't find the container with id d158d1e519bbb2ff90c3d646a9742f85fcef0405be4b0366062739bf1356bf10 Jul 05 19:30:03.087635 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.087607 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb49904b6_fdf6_47d2_8235_bd1927aff2f9.slice/crio-036ff2921c5a892d99d496f5d71421b387deaa0a7953f6acd5909f143f4533ff WatchSource:0}: Error finding container 036ff2921c5a892d99d496f5d71421b387deaa0a7953f6acd5909f143f4533ff: Status 404 returned error can't find the container with id 036ff2921c5a892d99d496f5d71421b387deaa0a7953f6acd5909f143f4533ff Jul 05 19:30:03.088556 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.088487 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c654201_8758_4ce8_9b5c_0f6d47f62e8f.slice/crio-0cb556b56857aaac3b62816799bed2d6eb1f9ae2f3701780be99c72f1ad32c65 WatchSource:0}: Error finding container 0cb556b56857aaac3b62816799bed2d6eb1f9ae2f3701780be99c72f1ad32c65: Status 404 returned error can't find the container with id 0cb556b56857aaac3b62816799bed2d6eb1f9ae2f3701780be99c72f1ad32c65 Jul 05 19:30:03.091768 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.091744 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1049dfed_1d3f_499e_9c9b_ac4175a7fd12.slice/crio-2b1806b28eec2c073a293d8a2e7433ac096bf8bde8277aeb8f9d408ad428d4de WatchSource:0}: Error finding container 2b1806b28eec2c073a293d8a2e7433ac096bf8bde8277aeb8f9d408ad428d4de: Status 404 returned error can't find the container with id 2b1806b28eec2c073a293d8a2e7433ac096bf8bde8277aeb8f9d408ad428d4de Jul 05 19:30:03.092541 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.092503 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94267a0b_11cf_4f85_8c61_6c7b4bd088ab.slice/crio-a2c3143fa8f9cae61171f2077ba52fb707d81ce7ae3292867c9811af3ef2ddf9 WatchSource:0}: Error finding container a2c3143fa8f9cae61171f2077ba52fb707d81ce7ae3292867c9811af3ef2ddf9: Status 404 returned error can't find the container with id a2c3143fa8f9cae61171f2077ba52fb707d81ce7ae3292867c9811af3ef2ddf9 Jul 05 19:30:03.093451 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.093433 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26ef5e5_d807_44d7_9d12_905c17073d16.slice/crio-b7699fe35d05559dc85cbaf83496967629da9e37a0d5bfa98ba7b0681cb7f453 WatchSource:0}: Error finding container b7699fe35d05559dc85cbaf83496967629da9e37a0d5bfa98ba7b0681cb7f453: Status 404 returned error can't find the container with id b7699fe35d05559dc85cbaf83496967629da9e37a0d5bfa98ba7b0681cb7f453 Jul 05 19:30:03.095211 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.094936 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd679826_cb58_4de2_83b2_378bd372dbd0.slice/crio-d5598f2b890a92fd1b3c32078e151df637745f71f228b71bb229dd15ae86e017 WatchSource:0}: Error finding container d5598f2b890a92fd1b3c32078e151df637745f71f228b71bb229dd15ae86e017: Status 404 returned error can't find the container with id d5598f2b890a92fd1b3c32078e151df637745f71f228b71bb229dd15ae86e017 Jul 05 19:30:03.095546 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.095476 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8d4538_2d34_496f_8aec_a84417bda656.slice/crio-ebc131d4de23b55a205601dc9f2f4ab6374662acb4ef5b988d8c04753f341557 WatchSource:0}: Error finding container ebc131d4de23b55a205601dc9f2f4ab6374662acb4ef5b988d8c04753f341557: Status 404 returned error can't find the container with id ebc131d4de23b55a205601dc9f2f4ab6374662acb4ef5b988d8c04753f341557 Jul 05 19:30:03.097941 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:03.097917 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c5b67a_a46d_45c7_a3a1_03de771be4fc.slice/crio-93e2950518bb131a4a1285ee83f129d86d83a35db5fefb2b053cb354508a7129 WatchSource:0}: Error finding container 93e2950518bb131a4a1285ee83f129d86d83a35db5fefb2b053cb354508a7129: Status 404 returned error can't find the container with id 93e2950518bb131a4a1285ee83f129d86d83a35db5fefb2b053cb354508a7129 Jul 05 19:30:03.585964 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.585923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:03.586600 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.586174 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:03.586600 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.586250 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.586228576 +0000 UTC m=+6.042012013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:03.687441 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.686747 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:03.687441 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.686954 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:03.687441 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.687018 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.686998951 +0000 UTC m=+6.142782366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:03.788174 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:03.788133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:03.788370 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.788350 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:03.788458 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.788378 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:03.788458 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.788394 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:03.788567 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:03.788459 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.788438063 +0000 UTC m=+6.244221477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:04.012586 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.012492 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 19:25:02 +0000 UTC" deadline="2028-01-18 02:18:31.200935934 +0000 UTC" Jul 05 19:30:04.012586 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.012538 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13470h48m27.188403423s" Jul 05 19:30:04.055441 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.055410 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:04.055614 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:04.055534 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:04.056066 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.056044 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:04.056210 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:04.056179 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:04.078388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.078359 2581 generic.go:356] "Generic (PLEG): container finished" podID="9abece3269131efc7d2a5033be560fe2" containerID="bcae9338e87d4cbaa087cbf462fc5bf282e6c2c06608540c672787f0fb38c742" exitCode=0 Jul 05 19:30:04.078525 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.078452 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"9abece3269131efc7d2a5033be560fe2","Type":"ContainerDied","Data":"bcae9338e87d4cbaa087cbf462fc5bf282e6c2c06608540c672787f0fb38c742"} Jul 05 19:30:04.095999 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.095967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26dbz" event={"ID":"dd679826-cb58-4de2-83b2-378bd372dbd0","Type":"ContainerStarted","Data":"d5598f2b890a92fd1b3c32078e151df637745f71f228b71bb229dd15ae86e017"} Jul 05 19:30:04.116661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.116623 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jdm75" event={"ID":"5a8d4538-2d34-496f-8aec-a84417bda656","Type":"ContainerStarted","Data":"ebc131d4de23b55a205601dc9f2f4ab6374662acb4ef5b988d8c04753f341557"} Jul 05 19:30:04.128188 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.128159 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-756dc" event={"ID":"94267a0b-11cf-4f85-8c61-6c7b4bd088ab","Type":"ContainerStarted","Data":"a2c3143fa8f9cae61171f2077ba52fb707d81ce7ae3292867c9811af3ef2ddf9"} Jul 05 19:30:04.152223 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.152190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" event={"ID":"1049dfed-1d3f-499e-9c9b-ac4175a7fd12","Type":"ContainerStarted","Data":"2b1806b28eec2c073a293d8a2e7433ac096bf8bde8277aeb8f9d408ad428d4de"} Jul 05 19:30:04.159155 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.159069 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4jzkw" event={"ID":"5c654201-8758-4ce8-9b5c-0f6d47f62e8f","Type":"ContainerStarted","Data":"0cb556b56857aaac3b62816799bed2d6eb1f9ae2f3701780be99c72f1ad32c65"} Jul 05 19:30:04.170656 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.170631 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkxjb" event={"ID":"c52aea1b-58e4-41ac-ac04-db4a90d9c794","Type":"ContainerStarted","Data":"d158d1e519bbb2ff90c3d646a9742f85fcef0405be4b0366062739bf1356bf10"} Jul 05 19:30:04.180970 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.180943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" event={"ID":"bd4ad6f0c2d4023e232d083fe9e47834","Type":"ContainerStarted","Data":"e822f6b45f0bbdf6d755307b52e320a5e845e441dff099fb326c2ca64f5ed0a5"} Jul 05 19:30:04.197704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.197675 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerStarted","Data":"93e2950518bb131a4a1285ee83f129d86d83a35db5fefb2b053cb354508a7129"} Jul 05 19:30:04.197894 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.197840 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" podStartSLOduration=3.197822668 podStartE2EDuration="3.197822668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:04.197714323 +0000 UTC m=+4.653497757" watchObservedRunningTime="2026-07-05 19:30:04.197822668 +0000 UTC m=+4.653606092" Jul 05 19:30:04.205546 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.205234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"b7699fe35d05559dc85cbaf83496967629da9e37a0d5bfa98ba7b0681cb7f453"} Jul 05 19:30:04.212740 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:04.212704 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vhv6b" event={"ID":"b49904b6-fdf6-47d2-8235-bd1927aff2f9","Type":"ContainerStarted","Data":"036ff2921c5a892d99d496f5d71421b387deaa0a7953f6acd5909f143f4533ff"} Jul 05 19:30:05.054789 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:05.054746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:05.055404 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.054903 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:05.252896 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:05.251195 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"9abece3269131efc7d2a5033be560fe2","Type":"ContainerStarted","Data":"1921a6210c2dee216b11819af13fa5207aa256c33db033ea9849f6664b5568d8"} Jul 05 19:30:05.605996 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:05.605928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:05.606232 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.606160 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:05.606232 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.606226 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.6062064 +0000 UTC m=+10.061989827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:05.706684 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:05.706637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:05.706930 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.706909 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:05.707023 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.706981 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.706960951 +0000 UTC m=+10.162744367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:05.807533 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:05.807494 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:05.807742 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.807725 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:05.807811 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.807751 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:05.807811 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.807764 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:05.807924 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:05.807825 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.807803441 +0000 UTC m=+10.263586850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:06.054529 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:06.054442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:06.054691 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:06.054606 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:06.055063 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:06.055041 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:06.055476 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:06.055163 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:07.054954 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:07.054917 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:07.055177 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:07.055097 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:08.058902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:08.058869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:08.059407 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:08.058993 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:08.059407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:08.059065 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:08.059407 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:08.059256 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:09.055355 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:09.055320 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:09.055547 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.055463 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:09.639497 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:09.639449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:09.639976 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.639712 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:09.639976 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.639780 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.639758904 +0000 UTC m=+18.095542336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:09.740542 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:09.740501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:09.740707 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.740634 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:09.740707 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.740689 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.740674404 +0000 UTC m=+18.196457813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:09.841298 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:09.841259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:09.841485 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.841465 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:09.841560 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.841496 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:09.841560 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.841509 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:09.841682 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:09.841572 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.841552383 +0000 UTC m=+18.297335795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:10.058290 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:10.058212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:10.058431 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:10.058354 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:10.058767 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:10.058749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:10.058863 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:10.058843 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:11.054732 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:11.054696 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:11.055231 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:11.054826 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:12.057983 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:12.057954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:12.058414 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:12.057954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:12.058414 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:12.058107 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:12.058414 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:12.058178 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:13.054854 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:13.054809 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:13.055078 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:13.054956 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:14.055294 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:14.055258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:14.055728 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:14.055258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:14.055728 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:14.055380 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:14.055728 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:14.055504 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:15.055144 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:15.055110 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:15.055337 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:15.055231 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:16.055267 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:16.055230 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:16.055466 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:16.055233 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:16.055466 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:16.055370 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:16.055466 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:16.055449 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:17.054575 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:17.054533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:17.054789 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.054658 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:17.704110 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:17.704074 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:17.704570 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.704241 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:17.704570 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.704311 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.704295573 +0000 UTC m=+34.160078985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:17.804710 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:17.804671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:17.804867 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.804842 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:17.804924 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.804909 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.804894025 +0000 UTC m=+34.260677437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:17.905090 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:17.905033 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:17.905280 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.905253 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:17.905280 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.905276 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:17.905392 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.905291 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:17.905392 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:17.905354 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.905335955 +0000 UTC m=+34.361119378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:18.055453 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:18.055364 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:18.055616 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:18.055372 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:18.055616 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:18.055507 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:18.055714 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:18.055618 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:19.054627 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:19.054592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:19.055036 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:19.054694 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:20.055777 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:20.055749 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:20.056211 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:20.055838 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:20.056211 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:20.055925 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:20.056211 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:20.056015 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:21.054771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:21.054731 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:21.054965 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:21.054866 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:22.055415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:22.055376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:22.055415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:22.055375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:22.055871 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:22.055505 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:22.055919 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:22.055892 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:23.054988 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.054680 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:23.055108 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:23.055086 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:23.290209 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.290177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vhv6b" event={"ID":"b49904b6-fdf6-47d2-8235-bd1927aff2f9","Type":"ContainerStarted","Data":"033e7e18db1ace2ab5235c989ee3167c570797f705a22b413a5643c9fc0da174"} Jul 05 19:30:23.291561 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.291539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26dbz" event={"ID":"dd679826-cb58-4de2-83b2-378bd372dbd0","Type":"ContainerStarted","Data":"e83aa150611ca13774011cee39a386d1f25822b64d849933bfd5ab8ee89f3a8b"} Jul 05 19:30:23.292810 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.292791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jdm75" event={"ID":"5a8d4538-2d34-496f-8aec-a84417bda656","Type":"ContainerStarted","Data":"f8dda8d42ebf82ad2878e8023a89af0599bc0d747e4490e61023ba81bb86f002"} Jul 05 19:30:23.294257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.294226 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-756dc" event={"ID":"94267a0b-11cf-4f85-8c61-6c7b4bd088ab","Type":"ContainerStarted","Data":"fe57f0fd8056aad34e52ac20ea18eaba767c9c8e3b5abd9e140094bb0c97b315"} Jul 05 19:30:23.295566 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.295545 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" event={"ID":"1049dfed-1d3f-499e-9c9b-ac4175a7fd12","Type":"ContainerStarted","Data":"244cd2939aff6fa03421ed8e6de78e8ed7849aa8ee3b43f4e6de43eed7398457"} Jul 05 19:30:23.297104 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.297032 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jkxjb" event={"ID":"c52aea1b-58e4-41ac-ac04-db4a90d9c794","Type":"ContainerStarted","Data":"08766cf82e8aa4a09471d7b34fb440427010c473ba863cab0be715ade52bdea7"} Jul 05 19:30:23.298937 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.298504 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="776701b2d3938fe7034849ccfd1d5db95d549a560d1e01b99bcd8722a4e59849" exitCode=0 Jul 05 19:30:23.298937 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.298578 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"776701b2d3938fe7034849ccfd1d5db95d549a560d1e01b99bcd8722a4e59849"} Jul 05 19:30:23.304592 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.304572 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:30:23.304982 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.304958 2581 generic.go:356] "Generic (PLEG): container finished" podID="b26ef5e5-d807-44d7-9d12-905c17073d16" containerID="8a297365f1484d966ae8a53baf805cc663a2a97b9f89901d1f42de5b44a6e2a3" exitCode=1 Jul 05 19:30:23.305092 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"ca5dc076dd136c394f99c4e43c3df3d851561bf4d35122555ec2319bc565463d"} Jul 05 19:30:23.305166 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305141 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"7bdcb20466b31223ca232af275e1b77ca04b0185f5a23127f7b408dc4b79db07"} Jul 05 19:30:23.305256 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305204 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"2dfdf140680a76e00db9a238262e751be5a8df89bbbf5a5836598de8f417bf1f"} Jul 05 19:30:23.305256 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"84f8ee5e43c323818188784c60ce94637dfef06afe213b49399b218cb3bf237b"} Jul 05 19:30:23.305329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305294 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerDied","Data":"8a297365f1484d966ae8a53baf805cc663a2a97b9f89901d1f42de5b44a6e2a3"} Jul 05 19:30:23.305329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.305325 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"cde7841141293123db9156017e58e61490f856a23ddcca7f71c12aaf81f1e937"} Jul 05 19:30:23.306501 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.306453 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" podStartSLOduration=22.306437513 podStartE2EDuration="22.306437513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:05.269215975 +0000 UTC m=+5.724999410" watchObservedRunningTime="2026-07-05 19:30:23.306437513 +0000 UTC m=+23.762220944" Jul 05 19:30:23.306953 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.306918 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vhv6b" podStartSLOduration=11.965165277 podStartE2EDuration="23.306906438s" totalImagesPullingTime="11.341741161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.306314038 +0000 UTC m=+23.762097467" watchObservedRunningTime="2026-07-05 19:30:23.306906438 +0000 UTC m=+23.762689869" Jul 05 19:30:23.331931 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.331891 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jdm75" podStartSLOduration=3.966940175 podStartE2EDuration="23.331879041s" totalImagesPullingTime="19.364938866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.331540742 +0000 UTC m=+23.787324171" watchObservedRunningTime="2026-07-05 19:30:23.331879041 +0000 UTC m=+23.787662470" Jul 05 19:30:23.332092 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.331970 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jkxjb" podStartSLOduration=3.949322975 podStartE2EDuration="23.331963185s" totalImagesPullingTime="19.38264021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.318759705 +0000 UTC m=+23.774543135" watchObservedRunningTime="2026-07-05 19:30:23.331963185 +0000 UTC m=+23.787746616" Jul 05 19:30:23.365537 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:23.365458 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-756dc" podStartSLOduration=3.9433235140000003 podStartE2EDuration="23.365438783s" totalImagesPullingTime="19.422115269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.365422902 +0000 UTC m=+23.821206343" watchObservedRunningTime="2026-07-05 19:30:23.365438783 +0000 UTC m=+23.821222213" Jul 05 19:30:24.024754 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.024731 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 05 19:30:24.035025 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.034945 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-05T19:30:24.024748956Z","UUID":"a5d742ad-aba4-4c60-888d-f9c38254e094","Handler":null,"Name":"","Endpoint":""} Jul 05 19:30:24.036189 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.036174 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 05 19:30:24.036261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.036207 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 05 19:30:24.054966 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.054944 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:24.054966 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.054954 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:24.055123 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:24.055028 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:24.055205 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:24.055186 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:24.309034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.308962 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" event={"ID":"1049dfed-1d3f-499e-9c9b-ac4175a7fd12","Type":"ContainerStarted","Data":"23997d6aaf73986cea8dd5c2bc125c18f1704dd2126cd3506c6a9686202d8b37"} Jul 05 19:30:24.310740 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.310680 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4jzkw" event={"ID":"5c654201-8758-4ce8-9b5c-0f6d47f62e8f","Type":"ContainerStarted","Data":"8d71ebecfd9cf50a374b969dc24fcbaf27b2b37761f39ad5a2894cb3515b640d"} Jul 05 19:30:24.326815 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.326773 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26dbz" podStartSLOduration=4.958924609 podStartE2EDuration="24.326755105s" totalImagesPullingTime="19.367830496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.382683051 +0000 UTC m=+23.838466481" watchObservedRunningTime="2026-07-05 19:30:24.326755105 +0000 UTC m=+24.782538535" Jul 05 19:30:24.327083 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:24.327039 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4jzkw" podStartSLOduration=4.954633811 podStartE2EDuration="24.327034281s" totalImagesPullingTime="19.37240047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:24.326532972 +0000 UTC m=+24.782316402" watchObservedRunningTime="2026-07-05 19:30:24.327034281 +0000 UTC m=+24.782817712" Jul 05 19:30:25.055216 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:25.055183 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:25.055486 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:25.055315 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:25.478833 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:25.478585 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:25.479296 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:25.479275 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:26.055297 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.055268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:26.055471 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.055266 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:26.055471 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:26.055380 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:26.055471 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:26.055462 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:26.322742 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.322659 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:30:26.323113 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.323088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"e33d3f11cc1ff48bfbf922f3c8bea46428efdba3fa9f5193aa6124222809befd"} Jul 05 19:30:26.325416 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.325350 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" event={"ID":"1049dfed-1d3f-499e-9c9b-ac4175a7fd12","Type":"ContainerStarted","Data":"2bf9ab63c3fd9482cebcdd96596322767bbee81817f6775427af6365965f67f7"} Jul 05 19:30:26.325827 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.325623 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:26.326195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.326170 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vhv6b" Jul 05 19:30:26.343324 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:26.343277 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v9flv" podStartSLOduration=4.639808728 podStartE2EDuration="26.343262324s" totalImagesPullingTime="21.703453596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:26.343016742 +0000 UTC m=+26.798800166" watchObservedRunningTime="2026-07-05 19:30:26.343262324 +0000 UTC m=+26.799045753" Jul 05 19:30:27.054828 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:27.054791 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:27.055422 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:27.054925 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:28.055228 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.054945 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:28.055228 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.055009 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:28.055778 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:28.055242 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:28.055778 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:28.055314 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:28.331491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.331426 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="aadbd4e8ac9811960c1c50cd2e941239235e9a7887fa97d55da87e61f379c845" exitCode=0 Jul 05 19:30:28.331644 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.331510 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"aadbd4e8ac9811960c1c50cd2e941239235e9a7887fa97d55da87e61f379c845"} Jul 05 19:30:28.335130 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.335110 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:30:28.335519 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.335498 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"aab6c337aa5ef6c945430b238283b3f66c3bb019086bf7ba0bfbd7b3f850431e"} Jul 05 19:30:28.335876 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:28.335858 2581 scope.go:117] "RemoveContainer" containerID="8a297365f1484d966ae8a53baf805cc663a2a97b9f89901d1f42de5b44a6e2a3" Jul 05 19:30:29.055388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.055359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:29.056178 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:29.055467 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:29.341189 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.341137 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:30:29.341495 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.341475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" event={"ID":"b26ef5e5-d807-44d7-9d12-905c17073d16","Type":"ContainerStarted","Data":"018dcb669d5e4d00850effb534b694957a3b3c99fe222d60f6e511d71795604d"} Jul 05 19:30:29.341760 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.341734 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:29.341872 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.341807 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:29.341872 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.341850 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:29.357201 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.357179 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:29.357368 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.357350 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:30:29.369275 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.369235 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" podStartSLOduration=9.919980436 podStartE2EDuration="29.369221827s" totalImagesPullingTime="19.449241391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:29.367397546 +0000 UTC m=+29.823180975" watchObservedRunningTime="2026-07-05 19:30:29.369221827 +0000 UTC m=+29.825005265" Jul 05 19:30:29.738339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.738126 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6srrh"] Jul 05 19:30:29.738541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.738431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:29.738602 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:29.738545 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:29.741681 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.741655 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-72jl7"] Jul 05 19:30:29.741815 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.741754 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:29.741879 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:29.741856 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:29.742343 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.742306 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqkck"] Jul 05 19:30:29.742460 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:29.742411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:29.742531 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:29.742511 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:30.346002 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:30.345967 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="41fcfb1ff892a8202464283dd1e2992d306c62ebfb44df3ff1614ade485524b0" exitCode=0 Jul 05 19:30:30.346417 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:30.346066 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"41fcfb1ff892a8202464283dd1e2992d306c62ebfb44df3ff1614ade485524b0"} Jul 05 19:30:31.054409 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:31.054377 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:31.054409 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:31.054398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:31.054621 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:31.054426 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:31.054621 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:31.054493 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:31.054621 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:31.054532 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:31.054731 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:31.054619 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:32.352898 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:32.352860 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="a3f910ad5635cbe2f4c8364cafb864c6519c63c322dd02ac11e35b8bc33a4556" exitCode=0 Jul 05 19:30:32.353681 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:32.352942 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"a3f910ad5635cbe2f4c8364cafb864c6519c63c322dd02ac11e35b8bc33a4556"} Jul 05 19:30:33.054483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.054449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:33.054483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.054465 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:33.054483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.054449 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:33.054775 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.054574 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:33.054775 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.054643 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:33.054775 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.054719 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:33.727147 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.727107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:33.727636 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.727267 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:33.727636 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.727335 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret podName:e3003f33-196f-47f9-ad7b-55f6f99c48eb nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.727316646 +0000 UTC m=+66.183100066 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret") pod "global-pull-secret-syncer-6srrh" (UID: "e3003f33-196f-47f9-ad7b-55f6f99c48eb") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:33.828137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.828103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:33.828291 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.828216 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:33.828291 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.828265 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs podName:b0b20ec4-573b-4f2f-9d2a-cf7412f1173e nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.828251387 +0000 UTC m=+66.284034795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs") pod "network-metrics-daemon-tqkck" (UID: "b0b20ec4-573b-4f2f-9d2a-cf7412f1173e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:33.929244 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:33.929211 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:33.929386 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.929330 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:33.929386 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.929346 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:33.929386 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.929356 2581 projected.go:196] Error preparing data for projected volume kube-api-access-8mnzk for pod openshift-network-diagnostics/network-check-target-72jl7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:33.929481 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:33.929402 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk podName:86e85607-5693-44f4-85d3-836cda7dce87 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.929387844 +0000 UTC m=+66.385171252 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mnzk" (UniqueName: "kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk") pod "network-check-target-72jl7" (UID: "86e85607-5693-44f4-85d3-836cda7dce87") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:35.054822 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:35.054786 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:35.055266 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:35.054921 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqkck" podUID="b0b20ec4-573b-4f2f-9d2a-cf7412f1173e" Jul 05 19:30:35.055266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:35.055008 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:35.055266 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:35.055120 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6srrh" podUID="e3003f33-196f-47f9-ad7b-55f6f99c48eb" Jul 05 19:30:35.055266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:35.055168 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:35.055266 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:35.055210 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-72jl7" podUID="86e85607-5693-44f4-85d3-836cda7dce87" Jul 05 19:30:36.306481 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.306256 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeReady" Jul 05 19:30:36.306885 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.306553 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 05 19:30:36.337667 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.337632 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:30:36.340431 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.340402 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.343016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.342996 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 05 19:30:36.343200 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.343178 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 05 19:30:36.343200 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.343192 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k29fx\"" Jul 05 19:30:36.343451 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.343430 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 05 19:30:36.348085 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.347828 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b4dq5"] Jul 05 19:30:36.348567 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.348535 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 05 19:30:36.349211 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.349195 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n6wtd"] Jul 05 19:30:36.349401 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.349383 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.350798 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.350779 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.351951 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.351933 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 05 19:30:36.352904 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.352887 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 05 19:30:36.353444 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.353428 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 05 19:30:36.353921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.353905 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8zj4k\"" Jul 05 19:30:36.354466 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.354192 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 05 19:30:36.354466 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.354308 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 05 19:30:36.354612 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.354469 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkl9h\"" Jul 05 19:30:36.356835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.356808 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:30:36.362342 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.362322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4dq5"] Jul 05 19:30:36.365112 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.365092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n6wtd"] Jul 05 19:30:36.447110 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447073 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a3896-1443-41de-aa31-60519c14caab-config-volume\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.447110 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447142 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6a3896-1443-41de-aa31-60519c14caab-tmp-dir\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447340 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6nl\" (UniqueName: \"kubernetes.io/projected/3afe59f0-b120-44a0-b841-4a26e694ce15-kube-api-access-gm6nl\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.447360 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447371 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3afe59f0-b120-44a0-b841-4a26e694ce15-cert\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447468 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447496 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9dh\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447569 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.447658 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.447610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2z85\" (UniqueName: \"kubernetes.io/projected/6a6a3896-1443-41de-aa31-60519c14caab-kube-api-access-r2z85\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.548704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.548704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9dh\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2z85\" (UniqueName: \"kubernetes.io/projected/6a6a3896-1443-41de-aa31-60519c14caab-kube-api-access-r2z85\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a3896-1443-41de-aa31-60519c14caab-config-volume\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.548915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548876 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6a3896-1443-41de-aa31-60519c14caab-tmp-dir\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548927 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548963 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.548992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6nl\" (UniqueName: \"kubernetes.io/projected/3afe59f0-b120-44a0-b841-4a26e694ce15-kube-api-access-gm6nl\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549087 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3afe59f0-b120-44a0-b841-4a26e694ce15-cert\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.549544 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549322 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6a3896-1443-41de-aa31-60519c14caab-tmp-dir\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.549544 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:36.549527 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:36.549634 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549564 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549634 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:36.549594 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.049575236 +0000 UTC m=+37.505358657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:36.549981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:36.549865 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:36.549981 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549867 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6a3896-1443-41de-aa31-60519c14caab-config-volume\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.549981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:36.549881 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:36.549981 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549884 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.549981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:36.549930 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.049916926 +0000 UTC m=+37.505700337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:36.550263 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.549994 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.553680 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.553659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.553786 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.553686 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3afe59f0-b120-44a0-b841-4a26e694ce15-cert\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.553952 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.553938 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.560826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.560757 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2z85\" (UniqueName: \"kubernetes.io/projected/6a6a3896-1443-41de-aa31-60519c14caab-kube-api-access-r2z85\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:36.561340 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.561323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9dh\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.561412 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.561360 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6nl\" (UniqueName: \"kubernetes.io/projected/3afe59f0-b120-44a0-b841-4a26e694ce15-kube-api-access-gm6nl\") pod \"ingress-canary-b4dq5\" (UID: \"3afe59f0-b120-44a0-b841-4a26e694ce15\") " pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:36.562438 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.562414 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:36.663748 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:36.663717 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b4dq5" Jul 05 19:30:37.052400 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.052370 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:37.052551 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.052429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:37.052551 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:37.052520 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:37.052551 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:37.052540 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:37.052551 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:37.052550 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:37.052673 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:37.052600 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:30:38.052576595 +0000 UTC m=+38.508360011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:37.052673 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:37.052621 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:38.052611582 +0000 UTC m=+38.508395026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:37.055475 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.055446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:30:37.055475 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.055471 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:30:37.055652 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.055484 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:30:37.058266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.058228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 19:30:37.058478 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.058312 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 05 19:30:37.058478 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.058337 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvhxs\"" Jul 05 19:30:37.058478 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.058472 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-698zg\"" Jul 05 19:30:37.058688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.058597 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 05 19:30:37.059609 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.059424 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 19:30:37.193659 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.193634 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b4dq5"] Jul 05 19:30:37.197098 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:30:37.197068 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afe59f0_b120_44a0_b841_4a26e694ce15.slice/crio-1f6c94e57dbcb37179b86a27d2a1508879b14f8c594786b0b1c6cd603a46e329 WatchSource:0}: Error finding container 1f6c94e57dbcb37179b86a27d2a1508879b14f8c594786b0b1c6cd603a46e329: Status 404 returned error can't find the container with id 1f6c94e57dbcb37179b86a27d2a1508879b14f8c594786b0b1c6cd603a46e329 Jul 05 19:30:37.371329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.371253 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="76c9fe5ff30b18d3b8672d3dd8977cde5dd2b09c02a5d539aabc9e467e5b7b85" exitCode=0 Jul 05 19:30:37.371670 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.371334 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"76c9fe5ff30b18d3b8672d3dd8977cde5dd2b09c02a5d539aabc9e467e5b7b85"} Jul 05 19:30:37.372506 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:37.372481 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4dq5" event={"ID":"3afe59f0-b120-44a0-b841-4a26e694ce15","Type":"ContainerStarted","Data":"1f6c94e57dbcb37179b86a27d2a1508879b14f8c594786b0b1c6cd603a46e329"} Jul 05 19:30:38.058547 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:38.058380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:38.058700 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:38.058586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:38.058700 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:38.058522 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:38.058700 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:38.058663 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:30:40.058648692 +0000 UTC m=+40.514432100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:38.058700 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:38.058697 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:38.058860 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:38.058706 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:38.058860 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:38.058747 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:40.05873468 +0000 UTC m=+40.514518088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:38.378951 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:38.378836 2581 generic.go:356] "Generic (PLEG): container finished" podID="45c5b67a-a46d-45c7-a3a1-03de771be4fc" containerID="4266ff5350b93965b2ceb97d0ccbb60720efb34a7fd5f44f1d677ea839b7af06" exitCode=0 Jul 05 19:30:38.378951 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:38.378902 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerDied","Data":"4266ff5350b93965b2ceb97d0ccbb60720efb34a7fd5f44f1d677ea839b7af06"} Jul 05 19:30:39.385816 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:39.385777 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" event={"ID":"45c5b67a-a46d-45c7-a3a1-03de771be4fc","Type":"ContainerStarted","Data":"e2f0b0259eade6291cb66d5d6d0369a2d17cf6efbfd7b1597b89ed6c425d00eb"} Jul 05 19:30:39.408838 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:39.408792 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6wzpw" podStartSLOduration=16.910162967 podStartE2EDuration="39.408776185s" totalImagesPullingTime="22.498613218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:39.407127872 +0000 UTC m=+39.862911303" watchObservedRunningTime="2026-07-05 19:30:39.408776185 +0000 UTC m=+39.864559615" Jul 05 19:30:40.074004 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:40.073974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:40.074195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:40.074029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:40.074195 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:40.074068 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:40.074195 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:40.074135 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:30:44.074120139 +0000 UTC m=+44.529903547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:40.074195 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:40.074149 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:40.074195 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:40.074165 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:40.074406 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:40.074219 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:44.074201813 +0000 UTC m=+44.529985223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:40.388966 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:40.388931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b4dq5" event={"ID":"3afe59f0-b120-44a0-b841-4a26e694ce15","Type":"ContainerStarted","Data":"9534c8bf9b3eb2823406bc9592d16412928a206d004e490e9556981489673509"} Jul 05 19:30:40.404714 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:40.404665 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b4dq5" podStartSLOduration=1.717554952 podStartE2EDuration="4.404649861s" totalImagesPullingTime="2.687094909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:40.404150093 +0000 UTC m=+40.859933526" watchObservedRunningTime="2026-07-05 19:30:40.404649861 +0000 UTC m=+40.860433290" Jul 05 19:30:44.101378 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:44.101340 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:44.101404 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:44.101477 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:44.101486 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:44.101491 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:44.101538 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:52.101522912 +0000 UTC m=+52.557306320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:44.101739 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:44.101550 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:30:52.101544787 +0000 UTC m=+52.557328195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:46.058423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.058397 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4ztfh"] Jul 05 19:30:46.086446 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.086418 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ztfh"] Jul 05 19:30:46.086581 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.086528 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.089033 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.089012 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 05 19:30:46.089174 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.089126 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 05 19:30:46.090429 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.090409 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 05 19:30:46.090515 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.090436 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 05 19:30:46.090515 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.090413 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rl664\"" Jul 05 19:30:46.218976 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.218948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.219161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.218986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7fx\" (UniqueName: \"kubernetes.io/projected/6b5ddf24-8134-439a-a999-946c99797e24-kube-api-access-zp7fx\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.219161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.219070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b5ddf24-8134-439a-a999-946c99797e24-crio-socket\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.219161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.219118 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.219161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.219136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b5ddf24-8134-439a-a999-946c99797e24-data-volume\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319568 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b5ddf24-8134-439a-a999-946c99797e24-crio-socket\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319568 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319568 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319565 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b5ddf24-8134-439a-a999-946c99797e24-data-volume\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319587 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7fx\" (UniqueName: \"kubernetes.io/projected/6b5ddf24-8134-439a-a999-946c99797e24-kube-api-access-zp7fx\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.319708 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6b5ddf24-8134-439a-a999-946c99797e24-crio-socket\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.319725 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.319793 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:46.819771881 +0000 UTC m=+47.275555303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : configmap "kube-rbac-proxy" not found Jul 05 19:30:46.319822 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.319813 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:46.819803078 +0000 UTC m=+47.275586499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.320133 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.319919 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b5ddf24-8134-439a-a999-946c99797e24-data-volume\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.329044 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.329020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7fx\" (UniqueName: \"kubernetes.io/projected/6b5ddf24-8134-439a-a999-946c99797e24-kube-api-access-zp7fx\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.823067 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.823023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.823236 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:46.823081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:46.823236 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.823176 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.823236 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.823194 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:46.823352 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.823240 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:47.823223097 +0000 UTC m=+48.279006518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.823352 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:46.823257 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:47.823245793 +0000 UTC m=+48.279029201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : configmap "kube-rbac-proxy" not found Jul 05 19:30:47.831123 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:47.831085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:47.831527 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:47.831140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:47.831527 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:47.831257 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:47.831527 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:47.831308 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:47.831527 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:47.831329 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:49.831309929 +0000 UTC m=+50.287093337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:47.831527 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:47.831362 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:49.831347844 +0000 UTC m=+50.287131255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : configmap "kube-rbac-proxy" not found Jul 05 19:30:47.832245 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:47.832218 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jdm75_5a8d4538-2d34-496f-8aec-a84417bda656/dns-node-resolver/0.log" Jul 05 19:30:49.232915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:49.232886 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkxjb_c52aea1b-58e4-41ac-ac04-db4a90d9c794/node-ca/0.log" Jul 05 19:30:49.831861 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:49.831832 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b4dq5_3afe59f0-b120-44a0-b841-4a26e694ce15/serve-healthcheck-canary/0.log" Jul 05 19:30:49.846028 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:49.845994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:49.846191 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:49.846143 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:49.846191 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:49.846162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:49.846361 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:49.846203 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:53.846183871 +0000 UTC m=+54.301967279 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : configmap "kube-rbac-proxy" not found Jul 05 19:30:49.846361 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:49.846280 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:49.846361 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:49.846342 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:53.846326595 +0000 UTC m=+54.302110008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:52.162770 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:52.162734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:52.162802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:52.162895 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:52.162903 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:52.162924 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76766848db-95jpq: secret "image-registry-tls" not found Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:52.162956 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls podName:6a6a3896-1443-41de-aa31-60519c14caab nodeName:}" failed. No retries permitted until 2026-07-05 19:31:08.162940934 +0000 UTC m=+68.618724342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls") pod "dns-default-n6wtd" (UID: "6a6a3896-1443-41de-aa31-60519c14caab") : secret "dns-default-metrics-tls" not found Jul 05 19:30:52.163207 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:52.162977 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls podName:e53e5034-d4a8-4e4a-8119-fdafea019068 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:08.162961662 +0000 UTC m=+68.618745074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls") pod "image-registry-76766848db-95jpq" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068") : secret "image-registry-tls" not found Jul 05 19:30:53.874512 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:53.874471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:53.874981 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:30:53.874532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:30:53.874981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:53.874649 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:53.874981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:53.874676 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:53.874981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:53.874747 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:01.874730706 +0000 UTC m=+62.330514113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : configmap "kube-rbac-proxy" not found Jul 05 19:30:53.874981 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:30:53.874763 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:01.874757266 +0000 UTC m=+62.330540715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:31:01.366870 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:01.366844 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5qtff" Jul 05 19:31:01.929959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:01.929921 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:01.929959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:01.929967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:01.930197 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:01.930081 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:31:01.930197 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:01.930138 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls podName:6b5ddf24-8134-439a-a999-946c99797e24 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:17.930123693 +0000 UTC m=+78.385907101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4ztfh" (UID: "6b5ddf24-8134-439a-a999-946c99797e24") : secret "insights-runtime-extractor-tls" not found Jul 05 19:31:01.930487 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:01.930470 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6b5ddf24-8134-439a-a999-946c99797e24-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:05.756759 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.756727 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:31:05.759452 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.759431 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 19:31:05.769933 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.769909 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3003f33-196f-47f9-ad7b-55f6f99c48eb-original-pull-secret\") pod \"global-pull-secret-syncer-6srrh\" (UID: \"e3003f33-196f-47f9-ad7b-55f6f99c48eb\") " pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:31:05.857913 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.857877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:31:05.860711 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.860692 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 19:31:05.871190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.871171 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0b20ec4-573b-4f2f-9d2a-cf7412f1173e-metrics-certs\") pod \"network-metrics-daemon-tqkck\" (UID: \"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e\") " pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:31:05.876605 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.876589 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvhxs\"" Jul 05 19:31:05.884817 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.884801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6srrh" Jul 05 19:31:05.884925 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.884802 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqkck" Jul 05 19:31:05.960073 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.959546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:31:05.962754 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.962507 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 05 19:31:05.975773 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.975559 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 05 19:31:05.986197 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:05.986148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnzk\" (UniqueName: \"kubernetes.io/projected/86e85607-5693-44f4-85d3-836cda7dce87-kube-api-access-8mnzk\") pod \"network-check-target-72jl7\" (UID: \"86e85607-5693-44f4-85d3-836cda7dce87\") " pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:31:06.034107 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.033985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqkck"] Jul 05 19:31:06.038420 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:06.038388 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b20ec4_573b_4f2f_9d2a_cf7412f1173e.slice/crio-e12bd967b7048a787c5feadd5b30f2a5962e571a347b30c30eabf299c70c331d WatchSource:0}: Error finding container e12bd967b7048a787c5feadd5b30f2a5962e571a347b30c30eabf299c70c331d: Status 404 returned error can't find the container with id e12bd967b7048a787c5feadd5b30f2a5962e571a347b30c30eabf299c70c331d Jul 05 19:31:06.049024 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.049003 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6srrh"] Jul 05 19:31:06.051751 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:06.051724 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3003f33_196f_47f9_ad7b_55f6f99c48eb.slice/crio-3301087f61762c2242e4b958748d56c58922f0de699002c0859a82ce50ff3e77 WatchSource:0}: Error finding container 3301087f61762c2242e4b958748d56c58922f0de699002c0859a82ce50ff3e77: Status 404 returned error can't find the container with id 3301087f61762c2242e4b958748d56c58922f0de699002c0859a82ce50ff3e77 Jul 05 19:31:06.181918 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.181889 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-698zg\"" Jul 05 19:31:06.190112 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.190099 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:31:06.307904 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.307838 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-72jl7"] Jul 05 19:31:06.310866 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:06.310836 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e85607_5693_44f4_85d3_836cda7dce87.slice/crio-a89f987bb80490311f5c585fdab854333a8f50aa064df018d71215596b2c5793 WatchSource:0}: Error finding container a89f987bb80490311f5c585fdab854333a8f50aa064df018d71215596b2c5793: Status 404 returned error can't find the container with id a89f987bb80490311f5c585fdab854333a8f50aa064df018d71215596b2c5793 Jul 05 19:31:06.455943 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.455899 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6srrh" event={"ID":"e3003f33-196f-47f9-ad7b-55f6f99c48eb","Type":"ContainerStarted","Data":"3301087f61762c2242e4b958748d56c58922f0de699002c0859a82ce50ff3e77"} Jul 05 19:31:06.457423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.457395 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqkck" event={"ID":"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e","Type":"ContainerStarted","Data":"e12bd967b7048a787c5feadd5b30f2a5962e571a347b30c30eabf299c70c331d"} Jul 05 19:31:06.458622 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:06.458598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-72jl7" event={"ID":"86e85607-5693-44f4-85d3-836cda7dce87","Type":"ContainerStarted","Data":"a89f987bb80490311f5c585fdab854333a8f50aa064df018d71215596b2c5793"} Jul 05 19:31:07.463976 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:07.463940 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqkck" event={"ID":"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e","Type":"ContainerStarted","Data":"7e5d19287d1b33675fde8d2f3394be1c2cfc0982834223eeaaa2f6b36b286dd6"} Jul 05 19:31:07.464453 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:07.464005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqkck" event={"ID":"b0b20ec4-573b-4f2f-9d2a-cf7412f1173e","Type":"ContainerStarted","Data":"2f80e7759a1579afb6c04f164719744a5636c2d9292847c58d4cdcd8931f31fd"} Jul 05 19:31:07.479880 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:07.479826 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tqkck" podStartSLOduration=66.475917184 podStartE2EDuration="1m7.479805643s" totalImagesPullingTime="1.003888459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:07.478535343 +0000 UTC m=+67.934318813" watchObservedRunningTime="2026-07-05 19:31:07.479805643 +0000 UTC m=+67.935589075" Jul 05 19:31:08.177921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.177880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:08.178117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.177956 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:31:08.180876 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.180849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6a3896-1443-41de-aa31-60519c14caab-metrics-tls\") pod \"dns-default-n6wtd\" (UID: \"6a6a3896-1443-41de-aa31-60519c14caab\") " pod="openshift-dns/dns-default-n6wtd" Jul 05 19:31:08.181413 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.181387 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"image-registry-76766848db-95jpq\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:08.456390 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.456313 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k29fx\"" Jul 05 19:31:08.464704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.464671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:08.477093 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.477069 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkl9h\"" Jul 05 19:31:08.485812 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.485794 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n6wtd" Jul 05 19:31:08.629635 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.629558 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:31:08.639862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:08.639833 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n6wtd"] Jul 05 19:31:09.662748 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:09.662716 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6a3896_1443_41de_aa31_60519c14caab.slice/crio-66d608dfb50c7b717555307c0fdccf4728602c6962aa674580a77920a61d00c3 WatchSource:0}: Error finding container 66d608dfb50c7b717555307c0fdccf4728602c6962aa674580a77920a61d00c3: Status 404 returned error can't find the container with id 66d608dfb50c7b717555307c0fdccf4728602c6962aa674580a77920a61d00c3 Jul 05 19:31:09.663543 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:09.663493 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53e5034_d4a8_4e4a_8119_fdafea019068.slice/crio-e12644aec3be61e0a9376e98c105f31219fc3521658212b2f44e4ab7d7c8cc85 WatchSource:0}: Error finding container e12644aec3be61e0a9376e98c105f31219fc3521658212b2f44e4ab7d7c8cc85: Status 404 returned error can't find the container with id e12644aec3be61e0a9376e98c105f31219fc3521658212b2f44e4ab7d7c8cc85 Jul 05 19:31:10.474845 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:10.474802 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76766848db-95jpq" event={"ID":"e53e5034-d4a8-4e4a-8119-fdafea019068","Type":"ContainerStarted","Data":"e12644aec3be61e0a9376e98c105f31219fc3521658212b2f44e4ab7d7c8cc85"} Jul 05 19:31:10.476189 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:10.476149 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6wtd" event={"ID":"6a6a3896-1443-41de-aa31-60519c14caab","Type":"ContainerStarted","Data":"66d608dfb50c7b717555307c0fdccf4728602c6962aa674580a77920a61d00c3"} Jul 05 19:31:11.123876 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.123852 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk"] Jul 05 19:31:11.137338 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.137312 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8"] Jul 05 19:31:11.137443 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.137424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.141069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.140863 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 05 19:31:11.141069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.140867 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 05 19:31:11.141069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.140891 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 05 19:31:11.141069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.140870 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j6z9x\"" Jul 05 19:31:11.141517 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.141502 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 05 19:31:11.148399 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.148378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk"] Jul 05 19:31:11.148399 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.148404 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8"] Jul 05 19:31:11.148545 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.148482 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.150954 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.150933 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 05 19:31:11.159221 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.159204 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:31:11.200878 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.200857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/509a8ac0-cb5d-4729-8edb-bd4347960617-klusterlet-config\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.200985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.200887 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/509a8ac0-cb5d-4729-8edb-bd4347960617-tmp\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.200985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.200925 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18d7a074-3608-49c3-ab20-354dc944c508-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.200985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.200976 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/509a8ac0-cb5d-4729-8edb-bd4347960617-kube-api-access-c7kzm\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.201127 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.200998 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2flw\" (UniqueName: \"kubernetes.io/projected/18d7a074-3608-49c3-ab20-354dc944c508-kube-api-access-z2flw\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.225946 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.225634 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8c5bcbb9-lwplr"] Jul 05 19:31:11.227659 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.227510 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.242476 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.242448 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8c5bcbb9-lwplr"] Jul 05 19:31:11.301877 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.301851 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-bound-sa-token\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.301993 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.301893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tw4\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-kube-api-access-v5tw4\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.301993 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.301924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/509a8ac0-cb5d-4729-8edb-bd4347960617-klusterlet-config\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.302132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.301989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18d7a074-3608-49c3-ab20-354dc944c508-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.302132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2flw\" (UniqueName: \"kubernetes.io/projected/18d7a074-3608-49c3-ab20-354dc944c508-kube-api-access-z2flw\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.302132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0db61f8-fdb0-43ef-9918-9bf89879c706-ca-trust-extracted\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-trusted-ca\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-tls\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-certificates\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/509a8ac0-cb5d-4729-8edb-bd4347960617-tmp\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.302372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302209 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-installation-pull-secrets\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-image-registry-private-configuration\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.302372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/509a8ac0-cb5d-4729-8edb-bd4347960617-kube-api-access-c7kzm\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.302926 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.302902 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/509a8ac0-cb5d-4729-8edb-bd4347960617-tmp\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.305706 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.305679 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/509a8ac0-cb5d-4729-8edb-bd4347960617-klusterlet-config\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.307780 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.307754 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18d7a074-3608-49c3-ab20-354dc944c508-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.313086 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.312716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2flw\" (UniqueName: \"kubernetes.io/projected/18d7a074-3608-49c3-ab20-354dc944c508-kube-api-access-z2flw\") pod \"managed-serviceaccount-addon-agent-d4f66c5cb-2mskk\" (UID: \"18d7a074-3608-49c3-ab20-354dc944c508\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.313086 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.312887 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kzm\" (UniqueName: \"kubernetes.io/projected/509a8ac0-cb5d-4729-8edb-bd4347960617-kube-api-access-c7kzm\") pod \"klusterlet-addon-workmgr-fcb7c7875-v69x8\" (UID: \"509a8ac0-cb5d-4729-8edb-bd4347960617\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.402898 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.402869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0db61f8-fdb0-43ef-9918-9bf89879c706-ca-trust-extracted\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403076 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.402901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-trusted-ca\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403076 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.402923 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-tls\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403076 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.402947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-certificates\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403076 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.402980 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-installation-pull-secrets\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403076 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.403012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-image-registry-private-configuration\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.403078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-bound-sa-token\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.403107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tw4\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-kube-api-access-v5tw4\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.403418 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.403367 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0db61f8-fdb0-43ef-9918-9bf89879c706-ca-trust-extracted\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.404564 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.404535 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-certificates\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.405336 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.405312 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0db61f8-fdb0-43ef-9918-9bf89879c706-trusted-ca\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.406398 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.406365 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-installation-pull-secrets\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.406498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.406438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-registry-tls\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.406951 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.406932 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0db61f8-fdb0-43ef-9918-9bf89879c706-image-registry-private-configuration\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.418259 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.418236 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-bound-sa-token\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.419041 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.419019 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tw4\" (UniqueName: \"kubernetes.io/projected/e0db61f8-fdb0-43ef-9918-9bf89879c706-kube-api-access-v5tw4\") pod \"image-registry-5d8c5bcbb9-lwplr\" (UID: \"e0db61f8-fdb0-43ef-9918-9bf89879c706\") " pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.461191 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.461166 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" Jul 05 19:31:11.470724 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.470553 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:11.484359 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.484330 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76766848db-95jpq" event={"ID":"e53e5034-d4a8-4e4a-8119-fdafea019068","Type":"ContainerStarted","Data":"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4"} Jul 05 19:31:11.484569 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.484553 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:11.488081 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.487132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-72jl7" event={"ID":"86e85607-5693-44f4-85d3-836cda7dce87","Type":"ContainerStarted","Data":"d9e82708e0e4afae4e6fd93acf29afe94c0b78f6988d3b078162c862a8906b0e"} Jul 05 19:31:11.488081 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.487641 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:31:11.490550 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.490106 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6srrh" event={"ID":"e3003f33-196f-47f9-ad7b-55f6f99c48eb","Type":"ContainerStarted","Data":"00b908368276edfe8a2252966a619395c3b888feff4822876b849694a17a128f"} Jul 05 19:31:11.505420 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.505368 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76766848db-95jpq" podStartSLOduration=45.505348126 podStartE2EDuration="45.505348126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:11.504255161 +0000 UTC m=+71.960038593" watchObservedRunningTime="2026-07-05 19:31:11.505348126 +0000 UTC m=+71.961131548" Jul 05 19:31:11.523047 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.522287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6srrh" podStartSLOduration=66.394936792 podStartE2EDuration="1m11.522267027s" totalImagesPullingTime="5.127330235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:11.521350985 +0000 UTC m=+71.977134418" watchObservedRunningTime="2026-07-05 19:31:11.522267027 +0000 UTC m=+71.978050458" Jul 05 19:31:11.538675 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.538475 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-72jl7" podStartSLOduration=66.672373864 podStartE2EDuration="1m11.538456155s" totalImagesPullingTime="4.866082291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:11.537323283 +0000 UTC m=+71.993106714" watchObservedRunningTime="2026-07-05 19:31:11.538456155 +0000 UTC m=+71.994239649" Jul 05 19:31:11.541242 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.540914 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:11.635123 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.635010 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk"] Jul 05 19:31:11.642415 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:11.642112 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d7a074_3608_49c3_ab20_354dc944c508.slice/crio-2305f06a16cff40452676038eec829b72a770724218e50ae5d2417ded26f3b16 WatchSource:0}: Error finding container 2305f06a16cff40452676038eec829b72a770724218e50ae5d2417ded26f3b16: Status 404 returned error can't find the container with id 2305f06a16cff40452676038eec829b72a770724218e50ae5d2417ded26f3b16 Jul 05 19:31:11.658662 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.658598 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8"] Jul 05 19:31:11.663864 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:11.663832 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509a8ac0_cb5d_4729_8edb_bd4347960617.slice/crio-4a409ed6adb5a8c35631916a59ecf87a2d7e994edc5acf9e955de425b86d8a4f WatchSource:0}: Error finding container 4a409ed6adb5a8c35631916a59ecf87a2d7e994edc5acf9e955de425b86d8a4f: Status 404 returned error can't find the container with id 4a409ed6adb5a8c35631916a59ecf87a2d7e994edc5acf9e955de425b86d8a4f Jul 05 19:31:11.716941 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:11.716873 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8c5bcbb9-lwplr"] Jul 05 19:31:11.721722 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:11.721695 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0db61f8_fdb0_43ef_9918_9bf89879c706.slice/crio-70f79f22e43e5793e4da356b879b864ea1b2de8e3c1d043e60f161a893daf506 WatchSource:0}: Error finding container 70f79f22e43e5793e4da356b879b864ea1b2de8e3c1d043e60f161a893daf506: Status 404 returned error can't find the container with id 70f79f22e43e5793e4da356b879b864ea1b2de8e3c1d043e60f161a893daf506 Jul 05 19:31:12.495888 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.495842 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" event={"ID":"18d7a074-3608-49c3-ab20-354dc944c508","Type":"ContainerStarted","Data":"2305f06a16cff40452676038eec829b72a770724218e50ae5d2417ded26f3b16"} Jul 05 19:31:12.496895 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.496873 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" event={"ID":"509a8ac0-cb5d-4729-8edb-bd4347960617","Type":"ContainerStarted","Data":"4a409ed6adb5a8c35631916a59ecf87a2d7e994edc5acf9e955de425b86d8a4f"} Jul 05 19:31:12.498304 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.498277 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" event={"ID":"e0db61f8-fdb0-43ef-9918-9bf89879c706","Type":"ContainerStarted","Data":"3da99e03191f91f449923f8c90fb8def14f941b9195a6bb15118998f0229f541"} Jul 05 19:31:12.498396 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.498327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" event={"ID":"e0db61f8-fdb0-43ef-9918-9bf89879c706","Type":"ContainerStarted","Data":"70f79f22e43e5793e4da356b879b864ea1b2de8e3c1d043e60f161a893daf506"} Jul 05 19:31:12.498594 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.498569 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:12.519236 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:12.519196 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" podStartSLOduration=1.519182534 podStartE2EDuration="1.519182534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:12.518506663 +0000 UTC m=+72.974290094" watchObservedRunningTime="2026-07-05 19:31:12.519182534 +0000 UTC m=+72.974965942" Jul 05 19:31:13.275554 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.275513 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th"] Jul 05 19:31:13.278743 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.278723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:13.281417 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.281388 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 05 19:31:13.281417 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.281408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-s4zwr\"" Jul 05 19:31:13.287567 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.287546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th"] Jul 05 19:31:13.317824 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.317798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25805d8e-fec2-40e7-ba31-78e90b70e517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7t9th\" (UID: \"25805d8e-fec2-40e7-ba31-78e90b70e517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:13.418642 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.418616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25805d8e-fec2-40e7-ba31-78e90b70e517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7t9th\" (UID: \"25805d8e-fec2-40e7-ba31-78e90b70e517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:13.421446 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.421418 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25805d8e-fec2-40e7-ba31-78e90b70e517-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-7t9th\" (UID: \"25805d8e-fec2-40e7-ba31-78e90b70e517\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:13.505739 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.505702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6wtd" event={"ID":"6a6a3896-1443-41de-aa31-60519c14caab","Type":"ContainerStarted","Data":"e404c189177fc4c4c6af7923d69c2b116e8138b00fdfb53d231b9a2efd813812"} Jul 05 19:31:13.506176 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.505768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n6wtd" event={"ID":"6a6a3896-1443-41de-aa31-60519c14caab","Type":"ContainerStarted","Data":"550db7f97094f985ec36155d37470e197b95fd8fcf5a108772d733944f4a515e"} Jul 05 19:31:13.506252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.506230 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n6wtd" Jul 05 19:31:13.524068 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.524002 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n6wtd" podStartSLOduration=34.661209401 podStartE2EDuration="37.523984293s" totalImagesPullingTime="2.862774892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:13.522893432 +0000 UTC m=+73.978676865" watchObservedRunningTime="2026-07-05 19:31:13.523984293 +0000 UTC m=+73.979767724" Jul 05 19:31:13.591462 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.591163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:13.736859 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:13.736830 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th"] Jul 05 19:31:14.417637 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:14.417604 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25805d8e_fec2_40e7_ba31_78e90b70e517.slice/crio-d4b2c49553d6241c13e2621707dc06881e1dc8c8ac15c3a4a15c3d9b1fbc2ab9 WatchSource:0}: Error finding container d4b2c49553d6241c13e2621707dc06881e1dc8c8ac15c3a4a15c3d9b1fbc2ab9: Status 404 returned error can't find the container with id d4b2c49553d6241c13e2621707dc06881e1dc8c8ac15c3a4a15c3d9b1fbc2ab9 Jul 05 19:31:14.510284 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:14.510247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" event={"ID":"25805d8e-fec2-40e7-ba31-78e90b70e517","Type":"ContainerStarted","Data":"d4b2c49553d6241c13e2621707dc06881e1dc8c8ac15c3a4a15c3d9b1fbc2ab9"} Jul 05 19:31:15.515975 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:15.515933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" event={"ID":"18d7a074-3608-49c3-ab20-354dc944c508","Type":"ContainerStarted","Data":"a2076ec51d3c182a9f12231da1150f2a93fa0dae0de0a1bb2997a62d2938f723"} Jul 05 19:31:15.532195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:15.532144 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d4f66c5cb-2mskk" podStartSLOduration=1.36838879 podStartE2EDuration="4.532126789s" totalImagesPullingTime="3.163737999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:15.530111378 +0000 UTC m=+75.985894810" watchObservedRunningTime="2026-07-05 19:31:15.532126789 +0000 UTC m=+75.987910219" Jul 05 19:31:16.520378 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:16.520335 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" event={"ID":"25805d8e-fec2-40e7-ba31-78e90b70e517","Type":"ContainerStarted","Data":"037e0af4b8f1ca4d9e1bb752b88183279b07d5c01634cca56ee89d9e9810de5a"} Jul 05 19:31:16.535633 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:16.535574 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" podStartSLOduration=2.351727283 podStartE2EDuration="3.535561029s" totalImagesPullingTime="1.183833746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:16.534547135 +0000 UTC m=+76.990330566" watchObservedRunningTime="2026-07-05 19:31:16.535561029 +0000 UTC m=+76.991344458" Jul 05 19:31:17.523982 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:17.523955 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:17.528645 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:17.528624 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-7t9th" Jul 05 19:31:17.954033 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:17.954003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:17.956386 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:17.956355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6b5ddf24-8134-439a-a999-946c99797e24-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4ztfh\" (UID: \"6b5ddf24-8134-439a-a999-946c99797e24\") " pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:18.198304 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:18.198278 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rl664\"" Jul 05 19:31:18.205719 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:18.205675 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4ztfh" Jul 05 19:31:18.332139 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:18.332029 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4ztfh"] Jul 05 19:31:18.334613 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:18.334592 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5ddf24_8134_439a_a999_946c99797e24.slice/crio-cb2319a8a8dd2963d7428a04373172f88ccf3557b5ef9cf41a8c82dff17c226c WatchSource:0}: Error finding container cb2319a8a8dd2963d7428a04373172f88ccf3557b5ef9cf41a8c82dff17c226c: Status 404 returned error can't find the container with id cb2319a8a8dd2963d7428a04373172f88ccf3557b5ef9cf41a8c82dff17c226c Jul 05 19:31:18.527851 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:18.527766 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ztfh" event={"ID":"6b5ddf24-8134-439a-a999-946c99797e24","Type":"ContainerStarted","Data":"cb2319a8a8dd2963d7428a04373172f88ccf3557b5ef9cf41a8c82dff17c226c"} Jul 05 19:31:22.544534 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:22.544501 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ztfh" event={"ID":"6b5ddf24-8134-439a-a999-946c99797e24","Type":"ContainerStarted","Data":"acfdd1ade8f3f6e0c8128cf9899e11a22b32656eda8f523afdf2a7abf5df7dc7"} Jul 05 19:31:23.550339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:23.550303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ztfh" event={"ID":"6b5ddf24-8134-439a-a999-946c99797e24","Type":"ContainerStarted","Data":"cca12882ad0cdb3c62c965d3f423c38072da425342ab4b03ea970093056afc22"} Jul 05 19:31:24.518796 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:24.518763 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n6wtd" Jul 05 19:31:25.560330 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.560250 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4ztfh" event={"ID":"6b5ddf24-8134-439a-a999-946c99797e24","Type":"ContainerStarted","Data":"bc67e06d86641c542a9023f80445ae3a840bb79b538e85032eabff236a1ffc0c"} Jul 05 19:31:25.586044 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.585989 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4ztfh" podStartSLOduration=32.82473046 podStartE2EDuration="39.585970981s" totalImagesPullingTime="6.761240521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:25.584845166 +0000 UTC m=+86.040628598" watchObservedRunningTime="2026-07-05 19:31:25.585970981 +0000 UTC m=+86.041754417" Jul 05 19:31:25.706580 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.706544 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8kp72"] Jul 05 19:31:25.709927 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.709905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.715179 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715135 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 05 19:31:25.715179 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715159 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 05 19:31:25.715179 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715174 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dkh7n\"" Jul 05 19:31:25.715383 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715196 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 05 19:31:25.715383 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715271 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 05 19:31:25.715473 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715460 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 05 19:31:25.715534 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.715519 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 05 19:31:25.818370 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-sys\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818370 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-wtmp\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818370 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-textfile\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-tls\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818492 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgqf4\" (UniqueName: \"kubernetes.io/projected/61302033-be14-4c7c-8834-2248c65f2119-kube-api-access-rgqf4\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818513 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-metrics-client-ca\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.818563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.818547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-root\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919797 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-wtmp\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-textfile\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-tls\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgqf4\" (UniqueName: \"kubernetes.io/projected/61302033-be14-4c7c-8834-2248c65f2119-kube-api-access-rgqf4\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.919936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-metrics-client-ca\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919956 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-wtmp\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.919973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-root\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-sys\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-sys\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920111 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61302033-be14-4c7c-8834-2248c65f2119-root\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920252 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-textfile\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920586 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920567 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-accelerators-collector-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.920665 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.920643 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61302033-be14-4c7c-8834-2248c65f2119-metrics-client-ca\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.922312 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.922288 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-tls\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.922429 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.922392 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61302033-be14-4c7c-8834-2248c65f2119-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:25.928102 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:25.928081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgqf4\" (UniqueName: \"kubernetes.io/projected/61302033-be14-4c7c-8834-2248c65f2119-kube-api-access-rgqf4\") pod \"node-exporter-8kp72\" (UID: \"61302033-be14-4c7c-8834-2248c65f2119\") " pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:26.022253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.022227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8kp72" Jul 05 19:31:26.033591 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:26.033569 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61302033_be14_4c7c_8834_2248c65f2119.slice/crio-77b655d2cce051115fc4de494ad692b39c62ae85acfeb4a8cf5cc08b67d9d462 WatchSource:0}: Error finding container 77b655d2cce051115fc4de494ad692b39c62ae85acfeb4a8cf5cc08b67d9d462: Status 404 returned error can't find the container with id 77b655d2cce051115fc4de494ad692b39c62ae85acfeb4a8cf5cc08b67d9d462 Jul 05 19:31:26.564630 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.564598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kp72" event={"ID":"61302033-be14-4c7c-8834-2248c65f2119","Type":"ContainerStarted","Data":"77b655d2cce051115fc4de494ad692b39c62ae85acfeb4a8cf5cc08b67d9d462"} Jul 05 19:31:26.726522 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.726484 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:31:26.730245 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.730220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.732938 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.732920 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 05 19:31:26.733083 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733042 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 05 19:31:26.733160 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.732920 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 05 19:31:26.733297 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733281 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 05 19:31:26.733650 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733628 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 05 19:31:26.733650 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733645 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 05 19:31:26.733835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733664 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 05 19:31:26.733835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733662 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 05 19:31:26.733934 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733847 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s72bd\"" Jul 05 19:31:26.733997 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.733934 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 05 19:31:26.743334 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.743308 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:31:26.827520 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827520 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827520 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827632 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827710 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827995 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827995 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827995 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.827995 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.827854 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qms\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929276 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929316 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929352 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929379 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929442 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:26.929466 2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:26.929536 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls podName:e5258e80-a611-4220-ae3f-6e5a90590dc1 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:27.429513429 +0000 UTC m=+87.885296839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1") : secret "alertmanager-main-tls" not found Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929615 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.929655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.930010 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.929674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qms\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.930185 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.930161 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.930385 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.930363 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.930890 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.930841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.932663 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.932623 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.932663 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.932650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.933142 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.933088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.933352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.933314 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.933450 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.933428 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.933515 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.933462 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.934736 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.934699 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.934914 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.934891 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:26.944534 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:26.944511 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qms\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:27.433464 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.433440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:27.435662 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.435646 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:27.568898 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.568864 2581 generic.go:356] "Generic (PLEG): container finished" podID="61302033-be14-4c7c-8834-2248c65f2119" containerID="0135d1e5572ca3c9bdb6d371fc7140d9616b2f83aaa2c50c6b8b146f661bc4b0" exitCode=0 Jul 05 19:31:27.569220 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.568927 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kp72" event={"ID":"61302033-be14-4c7c-8834-2248c65f2119","Type":"ContainerDied","Data":"0135d1e5572ca3c9bdb6d371fc7140d9616b2f83aaa2c50c6b8b146f661bc4b0"} Jul 05 19:31:27.641825 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.641633 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:31:27.794261 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:27.794230 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:31:27.797497 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:27.797470 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5258e80_a611_4220_ae3f_6e5a90590dc1.slice/crio-44abf62a5e289efc591e786eb8cc1da28d091213d380cb8abd125148abc24864 WatchSource:0}: Error finding container 44abf62a5e289efc591e786eb8cc1da28d091213d380cb8abd125148abc24864: Status 404 returned error can't find the container with id 44abf62a5e289efc591e786eb8cc1da28d091213d380cb8abd125148abc24864 Jul 05 19:31:28.574981 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:28.574937 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kp72" event={"ID":"61302033-be14-4c7c-8834-2248c65f2119","Type":"ContainerStarted","Data":"d90468f721b47210b5e72f10b8bd41627192dbbe2b23c364a46603eb8e8473b5"} Jul 05 19:31:28.575524 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:28.575009 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8kp72" event={"ID":"61302033-be14-4c7c-8834-2248c65f2119","Type":"ContainerStarted","Data":"966558fc9f467b2bfacd218010f7adff73cf3cbeb12f4e3643ff1e42b3fcdd21"} Jul 05 19:31:28.576329 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:28.576289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"44abf62a5e289efc591e786eb8cc1da28d091213d380cb8abd125148abc24864"} Jul 05 19:31:28.595506 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:28.595469 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8kp72" podStartSLOduration=2.4275002900000002 podStartE2EDuration="3.59545472s" totalImagesPullingTime="1.101072034s" totalInitContainerRuntime="66.882396ms" isStatefulPod=true podCreationTimestamp="2026-07-05 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:28.593118493 +0000 UTC m=+89.048901923" watchObservedRunningTime="2026-07-05 19:31:28.59545472 +0000 UTC m=+89.051238149" Jul 05 19:31:29.581646 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.581605 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b" exitCode=0 Jul 05 19:31:29.582047 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.581699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b"} Jul 05 19:31:29.955531 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.955500 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-558b58c774-6wrbd"] Jul 05 19:31:29.958609 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.958595 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:29.961309 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961289 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 05 19:31:29.961423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961295 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-56jxd\"" Jul 05 19:31:29.961491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961425 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 05 19:31:29.961491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961481 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7pbk0ndosp0pe\"" Jul 05 19:31:29.961632 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961558 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 05 19:31:29.961689 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.961673 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 05 19:31:29.966375 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:29.966354 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-558b58c774-6wrbd"] Jul 05 19:31:30.055352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055325 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-metrics-server-audit-profiles\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055460 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055380 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-client-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055460 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-tls\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055535 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bf99b85-5a74-4874-9776-95b090766765-audit-log\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055535 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vklbb\" (UniqueName: \"kubernetes.io/projected/8bf99b85-5a74-4874-9776-95b090766765-kube-api-access-vklbb\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055535 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.055637 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.055587 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-client-certs\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156157 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156134 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-tls\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156272 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156169 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bf99b85-5a74-4874-9776-95b090766765-audit-log\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156272 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vklbb\" (UniqueName: \"kubernetes.io/projected/8bf99b85-5a74-4874-9776-95b090766765-kube-api-access-vklbb\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156272 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-client-certs\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-metrics-server-audit-profiles\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-client-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.156628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.156598 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8bf99b85-5a74-4874-9776-95b090766765-audit-log\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.157028 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.157007 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.157576 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.157555 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8bf99b85-5a74-4874-9776-95b090766765-metrics-server-audit-profiles\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.158649 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.158625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-tls\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.158870 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.158849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-client-ca-bundle\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.158909 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.158892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8bf99b85-5a74-4874-9776-95b090766765-secret-metrics-server-client-certs\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.163688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.163666 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vklbb\" (UniqueName: \"kubernetes.io/projected/8bf99b85-5a74-4874-9776-95b090766765-kube-api-access-vklbb\") pod \"metrics-server-558b58c774-6wrbd\" (UID: \"8bf99b85-5a74-4874-9776-95b090766765\") " pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.267725 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.267654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:30.414706 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.414680 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-558b58c774-6wrbd"] Jul 05 19:31:30.419093 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:30.417912 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf99b85_5a74_4874_9776_95b090766765.slice/crio-82a25b2ec3a6518431b535cd33892ec504f664c6d799f422f5ec616675ccd9ea WatchSource:0}: Error finding container 82a25b2ec3a6518431b535cd33892ec504f664c6d799f422f5ec616675ccd9ea: Status 404 returned error can't find the container with id 82a25b2ec3a6518431b535cd33892ec504f664c6d799f422f5ec616675ccd9ea Jul 05 19:31:30.436084 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.436042 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc"] Jul 05 19:31:30.440641 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.440621 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:30.443262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.443238 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 05 19:31:30.443356 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.443302 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6fn7h\"" Jul 05 19:31:30.449258 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.449236 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc"] Jul 05 19:31:30.560924 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.560852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2wg\" (UniqueName: \"kubernetes.io/projected/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-kube-api-access-6j2wg\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:30.560924 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.560913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:30.585821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.585789 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" event={"ID":"8bf99b85-5a74-4874-9776-95b090766765","Type":"ContainerStarted","Data":"82a25b2ec3a6518431b535cd33892ec504f664c6d799f422f5ec616675ccd9ea"} Jul 05 19:31:30.661510 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.661479 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2wg\" (UniqueName: \"kubernetes.io/projected/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-kube-api-access-6j2wg\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:30.661655 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.661528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:30.661655 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:30.661635 2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 05 19:31:30.661763 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:30.661691 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert podName:c6f9d16b-cd62-44c3-82fe-0b27a89eb580 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:31.161673321 +0000 UTC m=+91.617456728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-6xnpc" (UID: "c6f9d16b-cd62-44c3-82fe-0b27a89eb580") : secret "monitoring-plugin-cert" not found Jul 05 19:31:30.670162 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:30.670135 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2wg\" (UniqueName: \"kubernetes.io/projected/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-kube-api-access-6j2wg\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:31.166264 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.166183 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:31.169098 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.169067 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c6f9d16b-cd62-44c3-82fe-0b27a89eb580-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6xnpc\" (UID: \"c6f9d16b-cd62-44c3-82fe-0b27a89eb580\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:31.351558 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.351533 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:31.491005 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.490977 2581 patch_prober.go:28] interesting pod/image-registry-76766848db-95jpq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 05 19:31:31.491117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.491024 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76766848db-95jpq" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 19:31:31.501521 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.501432 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc"] Jul 05 19:31:31.503938 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:31.503912 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f9d16b_cd62_44c3_82fe_0b27a89eb580.slice/crio-99973714b5dd2c63ae530d693e6dbbd92707f55cd49793b6e2f73face2a77d96 WatchSource:0}: Error finding container 99973714b5dd2c63ae530d693e6dbbd92707f55cd49793b6e2f73face2a77d96: Status 404 returned error can't find the container with id 99973714b5dd2c63ae530d693e6dbbd92707f55cd49793b6e2f73face2a77d96 Jul 05 19:31:31.546306 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.546273 2581 patch_prober.go:28] interesting pod/image-registry-5d8c5bcbb9-lwplr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 05 19:31:31.546407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.546319 2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" podUID="e0db61f8-fdb0-43ef-9918-9bf89879c706" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 19:31:31.593568 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.593539 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149"} Jul 05 19:31:31.594231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.594078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6"} Jul 05 19:31:31.594231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.594136 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955"} Jul 05 19:31:31.594231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.594175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae"} Jul 05 19:31:31.594960 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:31.594943 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" event={"ID":"c6f9d16b-cd62-44c3-82fe-0b27a89eb580","Type":"ContainerStarted","Data":"99973714b5dd2c63ae530d693e6dbbd92707f55cd49793b6e2f73face2a77d96"} Jul 05 19:31:32.142957 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.142926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:31:32.145924 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.145908 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.148502 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.148482 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 05 19:31:32.148649 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.148531 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 05 19:31:32.148837 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.148797 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 05 19:31:32.148984 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.148966 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 05 19:31:32.149161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.149114 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 05 19:31:32.149161 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.149127 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 05 19:31:32.149882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.149846 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ml255\"" Jul 05 19:31:32.150069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.150036 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 05 19:31:32.159681 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.159660 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:31:32.164199 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.163632 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 05 19:31:32.277744 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.277744 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.277959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.277959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.277959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277848 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.277959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.278178 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.277973 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtz2r\" (UniqueName: \"kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378732 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378732 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378733 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378956 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378755 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378956 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378956 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.378956 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtz2r\" (UniqueName: \"kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.379418 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.378968 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.379418 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.379302 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.380098 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.379841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.380098 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.379977 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.380833 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.380806 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.382335 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.382293 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.382835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.382817 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.387847 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.387814 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtz2r\" (UniqueName: \"kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r\") pod \"console-94dd884bc-2qbfm\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.468349 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.468084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:32.602851 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.602820 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" event={"ID":"8bf99b85-5a74-4874-9776-95b090766765","Type":"ContainerStarted","Data":"2b336efc81fc43bc0c76aa7807e56a350903f3f607a1cbd7615d699f0430c4f9"} Jul 05 19:31:32.608916 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.608891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0"} Jul 05 19:31:32.620922 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.620518 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" podStartSLOduration=2.011366824 podStartE2EDuration="3.620502093s" totalImagesPullingTime="1.609135269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:32.619430162 +0000 UTC m=+93.075213592" watchObservedRunningTime="2026-07-05 19:31:32.620502093 +0000 UTC m=+93.076285524" Jul 05 19:31:32.659536 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:32.658403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:31:32.661510 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:31:32.661471 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7be40d_ac7a_40ee_bceb_dc88030994bb.slice/crio-49b1ec0dfe3a93e250ab32fee49868c0495a65d1ddb7606876de81a838e8e0e4 WatchSource:0}: Error finding container 49b1ec0dfe3a93e250ab32fee49868c0495a65d1ddb7606876de81a838e8e0e4: Status 404 returned error can't find the container with id 49b1ec0dfe3a93e250ab32fee49868c0495a65d1ddb7606876de81a838e8e0e4 Jul 05 19:31:33.510383 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.510356 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8c5bcbb9-lwplr" Jul 05 19:31:33.615341 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.615290 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" event={"ID":"c6f9d16b-cd62-44c3-82fe-0b27a89eb580","Type":"ContainerStarted","Data":"de845ec8c68680100e70b53cac5f16fd660236fe6c7dfff67260195940e7f44d"} Jul 05 19:31:33.615761 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.615687 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:33.621266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.621243 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" Jul 05 19:31:33.621374 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.621286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerStarted","Data":"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db"} Jul 05 19:31:33.622877 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.622854 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94dd884bc-2qbfm" event={"ID":"ad7be40d-ac7a-40ee-bceb-dc88030994bb","Type":"ContainerStarted","Data":"49b1ec0dfe3a93e250ab32fee49868c0495a65d1ddb7606876de81a838e8e0e4"} Jul 05 19:31:33.632525 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.632439 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6xnpc" podStartSLOduration=2.022895576 podStartE2EDuration="3.632422146s" totalImagesPullingTime="1.60952657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:33.630962367 +0000 UTC m=+94.086745796" watchObservedRunningTime="2026-07-05 19:31:33.632422146 +0000 UTC m=+94.088205575" Jul 05 19:31:33.656826 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:33.656777 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.865585116 podStartE2EDuration="7.656763764s" totalImagesPullingTime="3.775290445s" totalInitContainerRuntime="15.888203ms" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:33.655769842 +0000 UTC m=+94.111553275" watchObservedRunningTime="2026-07-05 19:31:33.656763764 +0000 UTC m=+94.112547197" Jul 05 19:31:35.632904 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:35.632868 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94dd884bc-2qbfm" event={"ID":"ad7be40d-ac7a-40ee-bceb-dc88030994bb","Type":"ContainerStarted","Data":"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0"} Jul 05 19:31:35.650761 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:35.650713 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-94dd884bc-2qbfm" podStartSLOduration=0.879104992 podStartE2EDuration="3.65069459s" totalImagesPullingTime="2.771589598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:35.648527505 +0000 UTC m=+96.104310935" watchObservedRunningTime="2026-07-05 19:31:35.65069459 +0000 UTC m=+96.106478021" Jul 05 19:31:36.509612 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:36.509567 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76766848db-95jpq" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerName="registry" containerID="cri-o://fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4" gracePeriod=30 Jul 05 19:31:37.515575 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.515554 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:37.628425 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628344 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628425 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628398 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628447 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628493 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628514 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628627 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr9dh\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628697 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628728 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates\") pod \"e53e5034-d4a8-4e4a-8119-fdafea019068\" (UID: \"e53e5034-d4a8-4e4a-8119-fdafea019068\") " Jul 05 19:31:37.628985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.628962 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:31:37.629412 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.629387 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:31:37.631262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.631198 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:37.631262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.631239 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:31:37.631420 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.631337 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:37.631420 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.631364 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:31:37.631532 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.631514 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh" (OuterVolumeSpecName: "kube-api-access-nr9dh") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "kube-api-access-nr9dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:37.638307 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.638281 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e53e5034-d4a8-4e4a-8119-fdafea019068" (UID: "e53e5034-d4a8-4e4a-8119-fdafea019068"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:31:37.641885 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.641863 2581 generic.go:356] "Generic (PLEG): container finished" podID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerID="fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4" exitCode=0 Jul 05 19:31:37.641971 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.641904 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76766848db-95jpq" Jul 05 19:31:37.641971 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.641921 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76766848db-95jpq" event={"ID":"e53e5034-d4a8-4e4a-8119-fdafea019068","Type":"ContainerDied","Data":"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4"} Jul 05 19:31:37.641971 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.641967 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76766848db-95jpq" event={"ID":"e53e5034-d4a8-4e4a-8119-fdafea019068","Type":"ContainerDied","Data":"e12644aec3be61e0a9376e98c105f31219fc3521658212b2f44e4ab7d7c8cc85"} Jul 05 19:31:37.642105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.641989 2581 scope.go:117] "RemoveContainer" containerID="fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4" Jul 05 19:31:37.643367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.643345 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" event={"ID":"509a8ac0-cb5d-4729-8edb-bd4347960617","Type":"ContainerStarted","Data":"fbdaefb09d8a5a035d23b168fd76f056c0169059e2e8a817f991e48583fcc9ad"} Jul 05 19:31:37.643580 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.643560 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:37.645288 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.645271 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" Jul 05 19:31:37.650243 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.650225 2581 scope.go:117] "RemoveContainer" containerID="fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4" Jul 05 19:31:37.650494 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:31:37.650476 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4\": container with ID starting with fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4 not found: ID does not exist" containerID="fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4" Jul 05 19:31:37.650550 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.650501 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4"} err="failed to get container status \"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4\": rpc error: code = NotFound desc = could not find container \"fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4\": container with ID starting with fbf8194b7cb1e085906cadce76b4be9e32e5bee25edcff0226d5af0394c64ae4 not found: ID does not exist" Jul 05 19:31:37.662669 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.662634 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-fcb7c7875-v69x8" podStartSLOduration=0.889554941 podStartE2EDuration="26.662622126s" totalImagesPullingTime="25.773067185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:37.661076886 +0000 UTC m=+98.116860308" watchObservedRunningTime="2026-07-05 19:31:37.662622126 +0000 UTC m=+98.118405659" Jul 05 19:31:37.674546 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.674529 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:31:37.679578 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.679560 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76766848db-95jpq"] Jul 05 19:31:37.729889 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729864 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.729889 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729885 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-trusted-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729899 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9dh\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-kube-api-access-nr9dh\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729910 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-image-registry-private-configuration\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729919 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e53e5034-d4a8-4e4a-8119-fdafea019068-registry-certificates\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729928 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e53e5034-d4a8-4e4a-8119-fdafea019068-ca-trust-extracted\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729937 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e53e5034-d4a8-4e4a-8119-fdafea019068-installation-pull-secrets\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:37.730095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:37.729944 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e53e5034-d4a8-4e4a-8119-fdafea019068-bound-sa-token\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:31:38.060186 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:38.060152 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" path="/var/lib/kubelet/pods/e53e5034-d4a8-4e4a-8119-fdafea019068/volumes" Jul 05 19:31:42.468795 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:42.468766 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:42.469156 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:42.468826 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:42.473517 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:42.473497 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:42.664735 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:42.664710 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:31:43.508472 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:43.508443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-72jl7" Jul 05 19:31:50.268634 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:50.268605 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:31:50.269078 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:31:50.268653 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:32:10.272875 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:10.272847 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:32:10.277218 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:10.277194 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-558b58c774-6wrbd" Jul 05 19:32:45.860916 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.860881 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:45.861407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861236 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="alertmanager" containerID="cri-o://eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" gracePeriod=120 Jul 05 19:32:45.861407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861286 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-metric" containerID="cri-o://84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" gracePeriod=120 Jul 05 19:32:45.861407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861344 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="prom-label-proxy" containerID="cri-o://9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" gracePeriod=120 Jul 05 19:32:45.861628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861388 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy" containerID="cri-o://d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" gracePeriod=120 Jul 05 19:32:45.861628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861347 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-web" containerID="cri-o://1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" gracePeriod=120 Jul 05 19:32:45.861628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:45.861351 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="config-reloader" containerID="cri-o://6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" gracePeriod=120 Jul 05 19:32:46.893243 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893214 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" exitCode=0 Jul 05 19:32:46.893243 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893237 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" exitCode=0 Jul 05 19:32:46.893243 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893244 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" exitCode=0 Jul 05 19:32:46.893243 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893250 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" exitCode=0 Jul 05 19:32:46.893690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893283 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db"} Jul 05 19:32:46.893690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149"} Jul 05 19:32:46.893690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955"} Jul 05 19:32:46.893690 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:46.893404 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae"} Jul 05 19:32:47.104606 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.104586 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:47.249123 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249098 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249130 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249149 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249165 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69qms\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249184 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249215 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249275 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249309 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249357 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249391 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249420 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249448 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249494 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e5258e80-a611-4220-ae3f-6e5a90590dc1\" (UID: \"e5258e80-a611-4220-ae3f-6e5a90590dc1\") " Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249513 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:47.249541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249535 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:32:47.250129 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249759 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-metrics-client-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.250129 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.249777 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-main-db\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.252164 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252140 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:32:47.252287 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252201 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume" (OuterVolumeSpecName: "config-volume") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.252287 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252271 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:47.252682 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252565 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.252682 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252672 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.252982 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.252763 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.253101 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.253077 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms" (OuterVolumeSpecName: "kube-api-access-69qms") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "kube-api-access-69qms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:32:47.253608 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.253580 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out" (OuterVolumeSpecName: "config-out") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:32:47.254074 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.254031 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.257105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.257074 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.263548 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.263527 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config" (OuterVolumeSpecName: "web-config") pod "e5258e80-a611-4220-ae3f-6e5a90590dc1" (UID: "e5258e80-a611-4220-ae3f-6e5a90590dc1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:47.351070 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351023 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351070 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351071 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-volume\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351082 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qms\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-kube-api-access-69qms\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351090 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5258e80-a611-4220-ae3f-6e5a90590dc1-config-out\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351098 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5258e80-a611-4220-ae3f-6e5a90590dc1-tls-assets\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351106 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351116 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5258e80-a611-4220-ae3f-6e5a90590dc1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351124 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-web-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351132 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351140 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-cluster-tls-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.351190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.351149 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5258e80-a611-4220-ae3f-6e5a90590dc1-secret-alertmanager-main-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:32:47.900292 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900262 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" exitCode=0 Jul 05 19:32:47.900292 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900287 2581 generic.go:356] "Generic (PLEG): container finished" podID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerID="1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" exitCode=0 Jul 05 19:32:47.900749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0"} Jul 05 19:32:47.900749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900341 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:47.900749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900372 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6"} Jul 05 19:32:47.900749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900399 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5258e80-a611-4220-ae3f-6e5a90590dc1","Type":"ContainerDied","Data":"44abf62a5e289efc591e786eb8cc1da28d091213d380cb8abd125148abc24864"} Jul 05 19:32:47.900749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.900445 2581 scope.go:117] "RemoveContainer" containerID="9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" Jul 05 19:32:47.910712 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.910691 2581 scope.go:117] "RemoveContainer" containerID="84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" Jul 05 19:32:47.923345 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.923325 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:47.924985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.924949 2581 scope.go:117] "RemoveContainer" containerID="d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" Jul 05 19:32:47.926578 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.926560 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:47.931651 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.931636 2581 scope.go:117] "RemoveContainer" containerID="1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" Jul 05 19:32:47.938251 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.938232 2581 scope.go:117] "RemoveContainer" containerID="6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" Jul 05 19:32:47.945024 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.945006 2581 scope.go:117] "RemoveContainer" containerID="eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" Jul 05 19:32:47.950780 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.950761 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:47.951006 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.950995 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="alertmanager" Jul 05 19:32:47.951045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951008 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="alertmanager" Jul 05 19:32:47.951045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951026 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="config-reloader" Jul 05 19:32:47.951045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951032 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="config-reloader" Jul 05 19:32:47.951045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951039 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="init-config-reloader" Jul 05 19:32:47.951045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951044 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="init-config-reloader" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951077 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951085 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951091 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-metric" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951097 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-metric" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951102 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="prom-label-proxy" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951107 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="prom-label-proxy" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951112 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerName="registry" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951117 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerName="registry" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951121 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-web" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951126 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-web" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951163 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="alertmanager" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951171 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-web" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951177 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53e5034-d4a8-4e4a-8119-fdafea019068" containerName="registry" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951182 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="config-reloader" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951187 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951193 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="kube-rbac-proxy-metric" Jul 05 19:32:47.951232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951198 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" containerName="prom-label-proxy" Jul 05 19:32:47.951912 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.951887 2581 scope.go:117] "RemoveContainer" containerID="dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b" Jul 05 19:32:47.955665 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.955647 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:47.958271 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958253 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 05 19:32:47.958573 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958556 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s72bd\"" Jul 05 19:32:47.958637 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958607 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 05 19:32:47.958691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958657 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 05 19:32:47.958856 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958839 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 05 19:32:47.958957 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958924 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 05 19:32:47.958957 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958936 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 05 19:32:47.959141 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958961 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 05 19:32:47.959141 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.958928 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 05 19:32:47.959483 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.959465 2581 scope.go:117] "RemoveContainer" containerID="9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" Jul 05 19:32:47.959757 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.959733 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db\": container with ID starting with 9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db not found: ID does not exist" containerID="9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" Jul 05 19:32:47.959843 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.959769 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db"} err="failed to get container status \"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db\": rpc error: code = NotFound desc = could not find container \"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db\": container with ID starting with 9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db not found: ID does not exist" Jul 05 19:32:47.959843 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.959789 2581 scope.go:117] "RemoveContainer" containerID="84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" Jul 05 19:32:47.960099 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.960045 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0\": container with ID starting with 84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0 not found: ID does not exist" containerID="84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" Jul 05 19:32:47.960250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960106 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0"} err="failed to get container status \"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0\": rpc error: code = NotFound desc = could not find container \"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0\": container with ID starting with 84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0 not found: ID does not exist" Jul 05 19:32:47.960250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960129 2581 scope.go:117] "RemoveContainer" containerID="d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" Jul 05 19:32:47.960388 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.960369 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149\": container with ID starting with d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149 not found: ID does not exist" containerID="d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" Jul 05 19:32:47.960440 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960394 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149"} err="failed to get container status \"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149\": rpc error: code = NotFound desc = could not find container \"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149\": container with ID starting with d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149 not found: ID does not exist" Jul 05 19:32:47.960440 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960409 2581 scope.go:117] "RemoveContainer" containerID="1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" Jul 05 19:32:47.960653 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.960613 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6\": container with ID starting with 1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6 not found: ID does not exist" containerID="1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" Jul 05 19:32:47.960653 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960636 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6"} err="failed to get container status \"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6\": rpc error: code = NotFound desc = could not find container \"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6\": container with ID starting with 1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6 not found: ID does not exist" Jul 05 19:32:47.960653 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960649 2581 scope.go:117] "RemoveContainer" containerID="6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" Jul 05 19:32:47.960851 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.960834 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955\": container with ID starting with 6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955 not found: ID does not exist" containerID="6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" Jul 05 19:32:47.960907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960858 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955"} err="failed to get container status \"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955\": rpc error: code = NotFound desc = could not find container \"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955\": container with ID starting with 6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955 not found: ID does not exist" Jul 05 19:32:47.960907 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.960871 2581 scope.go:117] "RemoveContainer" containerID="eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" Jul 05 19:32:47.961114 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.961097 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae\": container with ID starting with eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae not found: ID does not exist" containerID="eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" Jul 05 19:32:47.961171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.961120 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae"} err="failed to get container status \"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae\": rpc error: code = NotFound desc = could not find container \"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae\": container with ID starting with eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae not found: ID does not exist" Jul 05 19:32:47.961171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.961133 2581 scope.go:117] "RemoveContainer" containerID="dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b" Jul 05 19:32:47.963565 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:32:47.961683 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b\": container with ID starting with dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b not found: ID does not exist" containerID="dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b" Jul 05 19:32:47.963565 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.961727 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b"} err="failed to get container status \"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b\": rpc error: code = NotFound desc = could not find container \"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b\": container with ID starting with dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b not found: ID does not exist" Jul 05 19:32:47.963565 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.961748 2581 scope.go:117] "RemoveContainer" containerID="9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db" Jul 05 19:32:47.966299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.966274 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 05 19:32:47.967373 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.966741 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db"} err="failed to get container status \"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db\": rpc error: code = NotFound desc = could not find container \"9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db\": container with ID starting with 9dd78b8deec3a092aab8b15e185c345d5563b2f17e036d1f234453601a3018db not found: ID does not exist" Jul 05 19:32:47.967373 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.966770 2581 scope.go:117] "RemoveContainer" containerID="84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0" Jul 05 19:32:47.967373 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.967088 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0"} err="failed to get container status \"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0\": rpc error: code = NotFound desc = could not find container \"84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0\": container with ID starting with 84ad4dec510e1bc8842c00cc63e2503ce3fc9fecd97405a895620dee9dbb02e0 not found: ID does not exist" Jul 05 19:32:47.967373 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.967133 2581 scope.go:117] "RemoveContainer" containerID="d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149" Jul 05 19:32:47.967622 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.967419 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149"} err="failed to get container status \"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149\": rpc error: code = NotFound desc = could not find container \"d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149\": container with ID starting with d1535d1eed50c693dc7fe4ac26440abf3f6834b29a8e2c4920f9ad52281ca149 not found: ID does not exist" Jul 05 19:32:47.967622 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.967446 2581 scope.go:117] "RemoveContainer" containerID="1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6" Jul 05 19:32:47.968100 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968078 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6"} err="failed to get container status \"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6\": rpc error: code = NotFound desc = could not find container \"1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6\": container with ID starting with 1c04d643e0c21c051557ced414c9534bcb6aaf20fad5a1c4080f19b8b607e5c6 not found: ID does not exist" Jul 05 19:32:47.968100 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968101 2581 scope.go:117] "RemoveContainer" containerID="6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955" Jul 05 19:32:47.968363 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968345 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955"} err="failed to get container status \"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955\": rpc error: code = NotFound desc = could not find container \"6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955\": container with ID starting with 6984d40854bb4e97cc8017d1e7519caa4e1a4a6aa7ba2a7944d72a5ea2787955 not found: ID does not exist" Jul 05 19:32:47.968421 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968364 2581 scope.go:117] "RemoveContainer" containerID="eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae" Jul 05 19:32:47.968605 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968583 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae"} err="failed to get container status \"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae\": rpc error: code = NotFound desc = could not find container \"eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae\": container with ID starting with eece23953947230b4cbb069db157f2be0ade3eec162583ec7d0c2496fc1ee0ae not found: ID does not exist" Jul 05 19:32:47.968660 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968606 2581 scope.go:117] "RemoveContainer" containerID="dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b" Jul 05 19:32:47.968868 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.968822 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b"} err="failed to get container status \"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b\": rpc error: code = NotFound desc = could not find container \"dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b\": container with ID starting with dc2365e7fec77ac78ac7be2d33cf94b47b8a464355bb71e124f0579bfacc2e8b not found: ID does not exist" Jul 05 19:32:47.971137 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:47.971119 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:48.055847 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.055809 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.055847 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.055852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.055874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.055958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.055985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056039 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056144 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxq7\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-kube-api-access-8qxq7\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056248 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.056519 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.056356 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.059166 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.059137 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5258e80-a611-4220-ae3f-6e5a90590dc1" path="/var/lib/kubelet/pods/e5258e80-a611-4220-ae3f-6e5a90590dc1/volumes" Jul 05 19:32:48.156978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.156952 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.156989 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157005 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157020 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157044 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxq7\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-kube-api-access-8qxq7\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157720 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.157720 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.157611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.158801 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.158776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.159226 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.159201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0289d-3382-497d-a542-fc02b87d7096-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.160613 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.160518 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.160795 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.160863 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.161081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161311 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.161157 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161413 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.161393 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.161540 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.161518 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0289d-3382-497d-a542-fc02b87d7096-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.162203 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.162182 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.162340 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.162322 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0289d-3382-497d-a542-fc02b87d7096-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.167165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.167148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxq7\" (UniqueName: \"kubernetes.io/projected/6dd0289d-3382-497d-a542-fc02b87d7096-kube-api-access-8qxq7\") pod \"alertmanager-main-0\" (UID: \"6dd0289d-3382-497d-a542-fc02b87d7096\") " pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.266477 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.266442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 05 19:32:48.401652 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.401628 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 05 19:32:48.403599 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:32:48.403571 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd0289d_3382_497d_a542_fc02b87d7096.slice/crio-6d9472cea703c4c5729604bd9e4502c671024af9013f3ab4d5ef77762db26f90 WatchSource:0}: Error finding container 6d9472cea703c4c5729604bd9e4502c671024af9013f3ab4d5ef77762db26f90: Status 404 returned error can't find the container with id 6d9472cea703c4c5729604bd9e4502c671024af9013f3ab4d5ef77762db26f90 Jul 05 19:32:48.905790 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.905762 2581 generic.go:356] "Generic (PLEG): container finished" podID="6dd0289d-3382-497d-a542-fc02b87d7096" containerID="07e742d4216cda60f11fdcf415fc7a905bda73d224ff9c2860627a1cbffdec3c" exitCode=0 Jul 05 19:32:48.906226 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.905847 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerDied","Data":"07e742d4216cda60f11fdcf415fc7a905bda73d224ff9c2860627a1cbffdec3c"} Jul 05 19:32:48.906226 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:48.905923 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"6d9472cea703c4c5729604bd9e4502c671024af9013f3ab4d5ef77762db26f90"} Jul 05 19:32:49.878932 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.878902 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77ff47d74f-dsv2n"] Jul 05 19:32:49.880928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.880912 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.883676 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.883646 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 05 19:32:49.883813 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.883715 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 05 19:32:49.883813 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.883766 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 05 19:32:49.883975 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.883832 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 05 19:32:49.883975 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.883904 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vcbm8\"" Jul 05 19:32:49.884122 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.884032 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 05 19:32:49.888603 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.888586 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 05 19:32:49.894294 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.894271 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77ff47d74f-dsv2n"] Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.915712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"41da291f64660729e71115f755e6e794d69b400eadf0ea712724b43beaa03ca8"} Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.915874 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"283797d79dd7a24e0e24fbecbdb35bab341a30490cf73a0251e2f83e614fd8b6"} Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.915957 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"a809d6fc25f305ea618732b0cae51b928c25d4157285d91c7e50eae4c29812c2"} Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.916033 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"7f4524fb6328123c17ad22dacc1e0c0b91efc2663984ac02e02f1f854adc876f"} Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.916123 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"0018ad25a24b883eefbbffee58bd1018bff7f7c1f7a68b086dfe2a7b96d6804e"} Jul 05 19:32:49.919559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.916198 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0289d-3382-497d-a542-fc02b87d7096","Type":"ContainerStarted","Data":"b96934fed7ccfb5efb114f1c250c8beafff9e10b1dda1615844064e1d6d6ca3c"} Jul 05 19:32:49.942740 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.942700 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.923497394 podStartE2EDuration="2.94268857s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.191176ms" isStatefulPod=false podCreationTimestamp="2026-07-05 19:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:32:49.941916225 +0000 UTC m=+170.397699683" watchObservedRunningTime="2026-07-05 19:32:49.94268857 +0000 UTC m=+170.398472001" Jul 05 19:32:49.973844 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.973813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-serving-certs-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.973980 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.973861 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-federate-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.973980 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.973882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-metrics-client-ca\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.973980 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.973967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjr2\" (UniqueName: \"kubernetes.io/projected/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-kube-api-access-nwjr2\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.974131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.974000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.974131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.974026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.974131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.974104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:49.974131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:49.974122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.074862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.074826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-serving-certs-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075021 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.074871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-federate-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075021 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.074890 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-metrics-client-ca\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075021 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.074959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjr2\" (UniqueName: \"kubernetes.io/projected/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-kube-api-access-nwjr2\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075021 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.074992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.075038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.075101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.075124 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.075745 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-metrics-client-ca\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.075879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.075823 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-serving-certs-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.076457 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.076430 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.077698 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.077675 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.077774 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.077734 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-federate-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.078138 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.078121 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-telemeter-client-tls\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.078450 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.078430 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-secret-telemeter-client\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.083585 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.083567 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjr2\" (UniqueName: \"kubernetes.io/projected/740a4233-c5e2-4d54-ae2b-6b93ee0e78b4-kube-api-access-nwjr2\") pod \"telemeter-client-77ff47d74f-dsv2n\" (UID: \"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4\") " pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.190624 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.190593 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" Jul 05 19:32:50.321808 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.321783 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77ff47d74f-dsv2n"] Jul 05 19:32:50.323783 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:32:50.323757 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740a4233_c5e2_4d54_ae2b_6b93ee0e78b4.slice/crio-663c909d834102c867096acaff36c30c72cb15484c7c82b207f62a1fdaccee3e WatchSource:0}: Error finding container 663c909d834102c867096acaff36c30c72cb15484c7c82b207f62a1fdaccee3e: Status 404 returned error can't find the container with id 663c909d834102c867096acaff36c30c72cb15484c7c82b207f62a1fdaccee3e Jul 05 19:32:50.921465 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:50.921399 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" event={"ID":"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4","Type":"ContainerStarted","Data":"663c909d834102c867096acaff36c30c72cb15484c7c82b207f62a1fdaccee3e"} Jul 05 19:32:52.932511 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:52.932467 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" event={"ID":"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4","Type":"ContainerStarted","Data":"3989afe6a38cf8cee6178467c920b856c41e461f1456d4d082a21c191036596b"} Jul 05 19:32:52.932882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:52.932535 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" event={"ID":"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4","Type":"ContainerStarted","Data":"a147c1068d61c75a7a669bcb99fe630beead1852117c7dfbe2e6e26a136ef890"} Jul 05 19:32:52.932882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:52.932565 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" event={"ID":"740a4233-c5e2-4d54-ae2b-6b93ee0e78b4","Type":"ContainerStarted","Data":"1cf3be1435f1b4b32e50dc75fad8cb929e705945b6494d7ef22b6611625b2caf"} Jul 05 19:32:52.959087 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:52.956243 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77ff47d74f-dsv2n" podStartSLOduration=1.97634814 podStartE2EDuration="3.956222522s" totalImagesPullingTime="1.979874382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:32:52.954522243 +0000 UTC m=+173.410305688" watchObservedRunningTime="2026-07-05 19:32:52.956222522 +0000 UTC m=+173.412005952" Jul 05 19:32:53.662487 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.662458 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:32:53.664500 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.664483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.677169 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.677140 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:32:53.807710 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807658 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807710 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807923 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807923 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807817 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpjs\" (UniqueName: \"kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807923 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807923 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807871 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.807923 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.807897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909110 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909123 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpjs\" (UniqueName: \"kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909222 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909462 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909285 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909920 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909895 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.909920 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909905 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.910130 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.909974 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.910172 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.910086 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.911722 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.911705 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.911857 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.911840 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.917173 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.917124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpjs\" (UniqueName: \"kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs\") pod \"console-77f66d9c56-24hqw\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:53.974355 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:53.974330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:32:54.102643 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:54.102616 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:32:54.104838 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:32:54.104812 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode794c406_a8be_4528_9508_00296f7e8ac2.slice/crio-6d0d7b02f78d8b70e779cdc08e6cacdb4ea894273e2b7f091f072f91437efc40 WatchSource:0}: Error finding container 6d0d7b02f78d8b70e779cdc08e6cacdb4ea894273e2b7f091f072f91437efc40: Status 404 returned error can't find the container with id 6d0d7b02f78d8b70e779cdc08e6cacdb4ea894273e2b7f091f072f91437efc40 Jul 05 19:32:54.941362 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:54.941329 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f66d9c56-24hqw" event={"ID":"e794c406-a8be-4528-9508-00296f7e8ac2","Type":"ContainerStarted","Data":"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337"} Jul 05 19:32:54.941560 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:54.941390 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f66d9c56-24hqw" event={"ID":"e794c406-a8be-4528-9508-00296f7e8ac2","Type":"ContainerStarted","Data":"6d0d7b02f78d8b70e779cdc08e6cacdb4ea894273e2b7f091f072f91437efc40"} Jul 05 19:32:54.959648 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:32:54.959584 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77f66d9c56-24hqw" podStartSLOduration=1.959568305 podStartE2EDuration="1.959568305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:32:54.95731984 +0000 UTC m=+175.413103272" watchObservedRunningTime="2026-07-05 19:32:54.959568305 +0000 UTC m=+175.415351735" Jul 05 19:33:03.974727 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:03.974680 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:33:03.975125 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:03.974753 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:33:03.979467 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:03.979446 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:33:04.986473 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:04.986440 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:33:05.054311 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:05.054282 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:33:30.075168 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.075100 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-94dd884bc-2qbfm" podUID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" containerName="console" containerID="cri-o://7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0" gracePeriod=15 Jul 05 19:33:30.324503 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.324475 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:33:30.416755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416727 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416777 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416812 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416835 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtz2r\" (UniqueName: \"kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416864 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416880 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.416910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.416902 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert\") pod \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\" (UID: \"ad7be40d-ac7a-40ee-bceb-dc88030994bb\") " Jul 05 19:33:30.417391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.417357 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:33:30.417391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.417374 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:33:30.417549 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.417370 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config" (OuterVolumeSpecName: "console-config") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:33:30.417549 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.417425 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:33:30.419132 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.419103 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:33:30.419579 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.419555 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:33:30.419579 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.419569 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r" (OuterVolumeSpecName: "kube-api-access-vtz2r") pod "ad7be40d-ac7a-40ee-bceb-dc88030994bb" (UID: "ad7be40d-ac7a-40ee-bceb-dc88030994bb"). InnerVolumeSpecName "kube-api-access-vtz2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:33:30.518131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518101 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518127 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518136 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-oauth-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518144 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtz2r\" (UniqueName: \"kubernetes.io/projected/ad7be40d-ac7a-40ee-bceb-dc88030994bb-kube-api-access-vtz2r\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518154 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-service-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518163 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-console-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:30.518352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:30.518171 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad7be40d-ac7a-40ee-bceb-dc88030994bb-oauth-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:33:31.075198 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.075161 2581 generic.go:356] "Generic (PLEG): container finished" podID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" containerID="7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0" exitCode=0 Jul 05 19:33:31.075608 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.075210 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94dd884bc-2qbfm" Jul 05 19:33:31.075608 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.075238 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94dd884bc-2qbfm" event={"ID":"ad7be40d-ac7a-40ee-bceb-dc88030994bb","Type":"ContainerDied","Data":"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0"} Jul 05 19:33:31.075608 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.075292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94dd884bc-2qbfm" event={"ID":"ad7be40d-ac7a-40ee-bceb-dc88030994bb","Type":"ContainerDied","Data":"49b1ec0dfe3a93e250ab32fee49868c0495a65d1ddb7606876de81a838e8e0e4"} Jul 05 19:33:31.075608 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.075314 2581 scope.go:117] "RemoveContainer" containerID="7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0" Jul 05 19:33:31.083930 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.083914 2581 scope.go:117] "RemoveContainer" containerID="7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0" Jul 05 19:33:31.084262 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:33:31.084243 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0\": container with ID starting with 7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0 not found: ID does not exist" containerID="7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0" Jul 05 19:33:31.084355 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.084271 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0"} err="failed to get container status \"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0\": rpc error: code = NotFound desc = could not find container \"7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0\": container with ID starting with 7da0849e872e943e98e97497772354d1baa387a471ddbf0a2bde56de8a76f8c0 not found: ID does not exist" Jul 05 19:33:31.096814 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.096793 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:33:31.100660 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:31.100639 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-94dd884bc-2qbfm"] Jul 05 19:33:32.059403 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:33:32.059367 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" path="/var/lib/kubelet/pods/ad7be40d-ac7a-40ee-bceb-dc88030994bb/volumes" Jul 05 19:34:06.116688 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.116657 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:34:06.117195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.116918 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" containerName="console" Jul 05 19:34:06.117195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.116928 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" containerName="console" Jul 05 19:34:06.117195 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.116975 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7be40d-ac7a-40ee-bceb-dc88030994bb" containerName="console" Jul 05 19:34:06.121242 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.121223 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.132968 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.132946 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:34:06.201412 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201376 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201494 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjbzt\" (UniqueName: \"kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.201583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.201575 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302799 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302973 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302973 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjbzt\" (UniqueName: \"kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302973 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302887 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302973 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.302973 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.302939 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.303258 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.303018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.303615 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.303588 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.303725 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.303625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.303725 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.303691 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.303877 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.303742 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.305494 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.305475 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.305548 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.305533 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.311847 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.311828 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjbzt\" (UniqueName: \"kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt\") pod \"console-6948588c6-77hhr\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.429529 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.429504 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:06.557279 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:06.557257 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:34:06.559772 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:34:06.559749 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f73c4d_b85f_4aba_86fa_ca0e8996acf4.slice/crio-d7bf5a1d4132f57d31b9e2668ac1a4da186cd1be55753b67876f271585617169 WatchSource:0}: Error finding container d7bf5a1d4132f57d31b9e2668ac1a4da186cd1be55753b67876f271585617169: Status 404 returned error can't find the container with id d7bf5a1d4132f57d31b9e2668ac1a4da186cd1be55753b67876f271585617169 Jul 05 19:34:07.199286 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:07.199255 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6948588c6-77hhr" event={"ID":"63f73c4d-b85f-4aba-86fa-ca0e8996acf4","Type":"ContainerStarted","Data":"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c"} Jul 05 19:34:07.199675 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:07.199300 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6948588c6-77hhr" event={"ID":"63f73c4d-b85f-4aba-86fa-ca0e8996acf4","Type":"ContainerStarted","Data":"d7bf5a1d4132f57d31b9e2668ac1a4da186cd1be55753b67876f271585617169"} Jul 05 19:34:07.223763 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:07.223721 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6948588c6-77hhr" podStartSLOduration=1.223705065 podStartE2EDuration="1.223705065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:34:07.221986046 +0000 UTC m=+247.677769467" watchObservedRunningTime="2026-07-05 19:34:07.223705065 +0000 UTC m=+247.679488495" Jul 05 19:34:16.429700 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:16.429648 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:16.430150 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:16.429726 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:16.434598 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:16.434573 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:17.237839 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:17.237810 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:34:17.285740 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:17.285704 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:34:42.306596 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.306538 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-77f66d9c56-24hqw" podUID="e794c406-a8be-4528-9508-00296f7e8ac2" containerName="console" containerID="cri-o://9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337" gracePeriod=15 Jul 05 19:34:42.544066 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.544027 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:34:42.700134 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgpjs\" (UniqueName: \"kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700143 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700166 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700187 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700467 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700351 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700467 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700453 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700565 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700499 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert\") pod \"e794c406-a8be-4528-9508-00296f7e8ac2\" (UID: \"e794c406-a8be-4528-9508-00296f7e8ac2\") " Jul 05 19:34:42.700618 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700592 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca" (OuterVolumeSpecName: "service-ca") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:34:42.700679 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700633 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config" (OuterVolumeSpecName: "console-config") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:34:42.700787 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700754 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-service-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.700787 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700778 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-console-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.700942 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.700889 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:34:42.701143 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.701040 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:34:42.702533 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.702498 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:34:42.702620 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.702561 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:34:42.702620 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.702596 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs" (OuterVolumeSpecName: "kube-api-access-dgpjs") pod "e794c406-a8be-4528-9508-00296f7e8ac2" (UID: "e794c406-a8be-4528-9508-00296f7e8ac2"). InnerVolumeSpecName "kube-api-access-dgpjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:34:42.801615 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.801589 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-oauth-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.801615 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.801610 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-oauth-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.801615 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.801618 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpjs\" (UniqueName: \"kubernetes.io/projected/e794c406-a8be-4528-9508-00296f7e8ac2-kube-api-access-dgpjs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.801783 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.801627 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e794c406-a8be-4528-9508-00296f7e8ac2-console-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:42.801783 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:42.801636 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e794c406-a8be-4528-9508-00296f7e8ac2-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:34:43.333539 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.333511 2581 generic.go:356] "Generic (PLEG): container finished" podID="e794c406-a8be-4528-9508-00296f7e8ac2" containerID="9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337" exitCode=0 Jul 05 19:34:43.333921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.333555 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f66d9c56-24hqw" Jul 05 19:34:43.333921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.333586 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f66d9c56-24hqw" event={"ID":"e794c406-a8be-4528-9508-00296f7e8ac2","Type":"ContainerDied","Data":"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337"} Jul 05 19:34:43.333921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.333625 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f66d9c56-24hqw" event={"ID":"e794c406-a8be-4528-9508-00296f7e8ac2","Type":"ContainerDied","Data":"6d0d7b02f78d8b70e779cdc08e6cacdb4ea894273e2b7f091f072f91437efc40"} Jul 05 19:34:43.333921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.333643 2581 scope.go:117] "RemoveContainer" containerID="9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337" Jul 05 19:34:43.342345 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.342330 2581 scope.go:117] "RemoveContainer" containerID="9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337" Jul 05 19:34:43.342576 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:34:43.342561 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337\": container with ID starting with 9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337 not found: ID does not exist" containerID="9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337" Jul 05 19:34:43.342618 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.342583 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337"} err="failed to get container status \"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337\": rpc error: code = NotFound desc = could not find container \"9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337\": container with ID starting with 9d24a40f2918322438b53ef390626b6d2975dad4895c3d1ad2957087d734a337 not found: ID does not exist" Jul 05 19:34:43.354090 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.354047 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:34:43.357697 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:43.357678 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f66d9c56-24hqw"] Jul 05 19:34:44.059490 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:44.059460 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e794c406-a8be-4528-9508-00296f7e8ac2" path="/var/lib/kubelet/pods/e794c406-a8be-4528-9508-00296f7e8ac2/volumes" Jul 05 19:34:59.965598 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:59.965568 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:34:59.970266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:59.970245 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 05 19:34:59.971222 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:34:59.971204 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:35:00.049517 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.049365 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx"] Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.049752 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e794c406-a8be-4528-9508-00296f7e8ac2" containerName="console" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.049766 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e794c406-a8be-4528-9508-00296f7e8ac2" containerName="console" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.049820 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e794c406-a8be-4528-9508-00296f7e8ac2" containerName="console" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.053798 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.056728 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-txjr4\"" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.056742 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.057966 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 05 19:35:00.062165 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.061695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx"] Jul 05 19:35:00.123071 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.123029 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.123229 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.123088 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6lg\" (UniqueName: \"kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.123229 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.123151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.223788 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.223702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.223788 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.223741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6lg\" (UniqueName: \"kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.223788 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.223770 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.224143 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.224126 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.224187 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.224147 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.246420 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.246383 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6lg\" (UniqueName: \"kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.363150 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.363115 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:00.492122 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:00.492097 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx"] Jul 05 19:35:00.494387 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:35:00.494356 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44875649_7601_4597_848e_aca1f97f1bfd.slice/crio-0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930 WatchSource:0}: Error finding container 0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930: Status 404 returned error can't find the container with id 0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930 Jul 05 19:35:01.405756 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:01.405712 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" event={"ID":"44875649-7601-4597-848e-aca1f97f1bfd","Type":"ContainerStarted","Data":"0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930"} Jul 05 19:35:08.436685 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:08.436651 2581 generic.go:356] "Generic (PLEG): container finished" podID="44875649-7601-4597-848e-aca1f97f1bfd" containerID="49580e4c4b1ae271724f0150fd3c2e74b1bc1be7a1f6cff0645dc506ba8cab6c" exitCode=0 Jul 05 19:35:08.437092 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:08.436742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" event={"ID":"44875649-7601-4597-848e-aca1f97f1bfd","Type":"ContainerDied","Data":"49580e4c4b1ae271724f0150fd3c2e74b1bc1be7a1f6cff0645dc506ba8cab6c"} Jul 05 19:35:08.437749 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:08.437734 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:35:11.450068 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:11.450017 2581 generic.go:356] "Generic (PLEG): container finished" podID="44875649-7601-4597-848e-aca1f97f1bfd" containerID="e05e0635d9c8e34db660baa0dd3f22fcbf897bfda0289e5fe33f2ee1464df9a4" exitCode=0 Jul 05 19:35:11.450571 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:11.450109 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" event={"ID":"44875649-7601-4597-848e-aca1f97f1bfd","Type":"ContainerDied","Data":"e05e0635d9c8e34db660baa0dd3f22fcbf897bfda0289e5fe33f2ee1464df9a4"} Jul 05 19:35:19.486843 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:19.486805 2581 generic.go:356] "Generic (PLEG): container finished" podID="44875649-7601-4597-848e-aca1f97f1bfd" containerID="731c12fee8e4a93b528f5b0bbdc3712d8f84e72716354abf8396ee24bc3ed7c5" exitCode=0 Jul 05 19:35:19.487193 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:19.486868 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" event={"ID":"44875649-7601-4597-848e-aca1f97f1bfd","Type":"ContainerDied","Data":"731c12fee8e4a93b528f5b0bbdc3712d8f84e72716354abf8396ee24bc3ed7c5"} Jul 05 19:35:20.623403 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.623373 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:20.689846 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.689817 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle\") pod \"44875649-7601-4597-848e-aca1f97f1bfd\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " Jul 05 19:35:20.690008 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.689888 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util\") pod \"44875649-7601-4597-848e-aca1f97f1bfd\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " Jul 05 19:35:20.690008 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.689921 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc6lg\" (UniqueName: \"kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg\") pod \"44875649-7601-4597-848e-aca1f97f1bfd\" (UID: \"44875649-7601-4597-848e-aca1f97f1bfd\") " Jul 05 19:35:20.690562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.690538 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle" (OuterVolumeSpecName: "bundle") pod "44875649-7601-4597-848e-aca1f97f1bfd" (UID: "44875649-7601-4597-848e-aca1f97f1bfd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:35:20.692166 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.692143 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg" (OuterVolumeSpecName: "kube-api-access-dc6lg") pod "44875649-7601-4597-848e-aca1f97f1bfd" (UID: "44875649-7601-4597-848e-aca1f97f1bfd"). InnerVolumeSpecName "kube-api-access-dc6lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:35:20.694656 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.694632 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util" (OuterVolumeSpecName: "util") pod "44875649-7601-4597-848e-aca1f97f1bfd" (UID: "44875649-7601-4597-848e-aca1f97f1bfd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:35:20.790673 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.790617 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6lg\" (UniqueName: \"kubernetes.io/projected/44875649-7601-4597-848e-aca1f97f1bfd-kube-api-access-dc6lg\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:35:20.790673 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.790641 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:35:20.790673 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:20.790650 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44875649-7601-4597-848e-aca1f97f1bfd-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:35:21.497196 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:21.497170 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" Jul 05 19:35:21.497196 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:21.497175 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409z2zx" event={"ID":"44875649-7601-4597-848e-aca1f97f1bfd","Type":"ContainerDied","Data":"0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930"} Jul 05 19:35:21.497397 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:21.497223 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c86d7a20fb79cba40eded77a8624caa1d0e64e3dce97390b3f7663d3c212930" Jul 05 19:35:28.061335 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061304 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm"] Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061561 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="extract" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061571 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="extract" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061584 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="pull" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061590 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="pull" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061599 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="util" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061604 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="util" Jul 05 19:35:28.061692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.061645 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="44875649-7601-4597-848e-aca1f97f1bfd" containerName="extract" Jul 05 19:35:28.063653 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.063639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.068200 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.068175 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jdtsf\"" Jul 05 19:35:28.068387 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.068363 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 05 19:35:28.068485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.068473 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:35:28.081663 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.081635 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm"] Jul 05 19:35:28.153202 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.153173 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc666\" (UniqueName: \"kubernetes.io/projected/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-kube-api-access-hc666\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.153343 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.153252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.253985 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.253950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.254177 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.254006 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc666\" (UniqueName: \"kubernetes.io/projected/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-kube-api-access-hc666\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.254390 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.254371 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.265640 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.265613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc666\" (UniqueName: \"kubernetes.io/projected/ea58ccf9-b41d-4e3d-9351-93a475b45e9d-kube-api-access-hc666\") pod \"cert-manager-operator-controller-manager-55c444bf86-pvhlm\" (UID: \"ea58ccf9-b41d-4e3d-9351-93a475b45e9d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.372164 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.372079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" Jul 05 19:35:28.513148 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.512985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm"] Jul 05 19:35:28.516183 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:35:28.516157 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea58ccf9_b41d_4e3d_9351_93a475b45e9d.slice/crio-e1f3dadd22affc4da2c4b8b11a81f0a4c243c2587b3e0fc4008726a10efd71b8 WatchSource:0}: Error finding container e1f3dadd22affc4da2c4b8b11a81f0a4c243c2587b3e0fc4008726a10efd71b8: Status 404 returned error can't find the container with id e1f3dadd22affc4da2c4b8b11a81f0a4c243c2587b3e0fc4008726a10efd71b8 Jul 05 19:35:28.525113 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:28.525085 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" event={"ID":"ea58ccf9-b41d-4e3d-9351-93a475b45e9d","Type":"ContainerStarted","Data":"e1f3dadd22affc4da2c4b8b11a81f0a4c243c2587b3e0fc4008726a10efd71b8"} Jul 05 19:35:31.539371 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:31.539337 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" event={"ID":"ea58ccf9-b41d-4e3d-9351-93a475b45e9d","Type":"ContainerStarted","Data":"f9348e931c8acfc2b1158629c750eebb1d23c53ae73516e17794bf34df6b50c6"} Jul 05 19:35:31.560529 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:31.560478 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-pvhlm" podStartSLOduration=1.409876881 podStartE2EDuration="3.560464867s" totalImagesPullingTime="2.150587986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:35:31.559129935 +0000 UTC m=+332.014913365" watchObservedRunningTime="2026-07-05 19:35:31.560464867 +0000 UTC m=+332.016248297" Jul 05 19:35:34.401202 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.401168 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xslx"] Jul 05 19:35:34.404403 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.404387 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.407694 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.407674 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 05 19:35:34.407813 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.407723 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 05 19:35:34.408877 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.408861 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5wf2v\"" Jul 05 19:35:34.416287 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.416267 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xslx"] Jul 05 19:35:34.509257 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.509237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.509364 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.509263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764mx\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-kube-api-access-764mx\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.610496 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.610469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.610625 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.610500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764mx\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-kube-api-access-764mx\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.619306 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.619279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.619306 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.619300 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-764mx\" (UniqueName: \"kubernetes.io/projected/29aa0cc1-de1a-49b1-916a-c192982a9977-kube-api-access-764mx\") pod \"cert-manager-cainjector-559cdd8c87-6xslx\" (UID: \"29aa0cc1-de1a-49b1-916a-c192982a9977\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.713495 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.713470 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" Jul 05 19:35:34.840181 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:34.840070 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6xslx"] Jul 05 19:35:34.842768 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:35:34.842740 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29aa0cc1_de1a_49b1_916a_c192982a9977.slice/crio-13ff0b2234e9dba68f341111560e77af98f6795faa35b9a52068d85b096ad1e5 WatchSource:0}: Error finding container 13ff0b2234e9dba68f341111560e77af98f6795faa35b9a52068d85b096ad1e5: Status 404 returned error can't find the container with id 13ff0b2234e9dba68f341111560e77af98f6795faa35b9a52068d85b096ad1e5 Jul 05 19:35:35.557039 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:35.557002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" event={"ID":"29aa0cc1-de1a-49b1-916a-c192982a9977","Type":"ContainerStarted","Data":"13ff0b2234e9dba68f341111560e77af98f6795faa35b9a52068d85b096ad1e5"} Jul 05 19:35:37.567563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:37.567531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" event={"ID":"29aa0cc1-de1a-49b1-916a-c192982a9977","Type":"ContainerStarted","Data":"82779bffe44e3a61727292c2ef9e43eb11b2a428ba0c356fafb4258fad8eec31"} Jul 05 19:35:37.585223 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:37.585161 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6xslx" podStartSLOduration=0.958936048 podStartE2EDuration="3.58514176s" totalImagesPullingTime="2.626205712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:35:37.583733863 +0000 UTC m=+338.039517293" watchObservedRunningTime="2026-07-05 19:35:37.58514176 +0000 UTC m=+338.040925191" Jul 05 19:35:51.951717 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:51.951646 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsqt9"] Jul 05 19:35:51.953674 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:51.953658 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:51.956210 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:51.956186 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cvbw2\"" Jul 05 19:35:51.967771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:51.967747 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsqt9"] Jul 05 19:35:52.054853 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.054798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxvk\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-kube-api-access-xrxvk\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.055044 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.054914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.155316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.155286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxvk\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-kube-api-access-xrxvk\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.155461 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.155358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.163730 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.163698 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-bound-sa-token\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.163845 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.163758 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxvk\" (UniqueName: \"kubernetes.io/projected/c7441598-99c7-4b22-b6ac-5e956b534700-kube-api-access-xrxvk\") pod \"cert-manager-55b4d954d5-vsqt9\" (UID: \"c7441598-99c7-4b22-b6ac-5e956b534700\") " pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.262325 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.262241 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vsqt9" Jul 05 19:35:52.389176 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.389152 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vsqt9"] Jul 05 19:35:52.391367 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:35:52.391342 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7441598_99c7_4b22_b6ac_5e956b534700.slice/crio-b3b93b9fb44254a512eeab5875387d31961b74e746a8478c15e7485ce59350e3 WatchSource:0}: Error finding container b3b93b9fb44254a512eeab5875387d31961b74e746a8478c15e7485ce59350e3: Status 404 returned error can't find the container with id b3b93b9fb44254a512eeab5875387d31961b74e746a8478c15e7485ce59350e3 Jul 05 19:35:52.627241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.627145 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vsqt9" event={"ID":"c7441598-99c7-4b22-b6ac-5e956b534700","Type":"ContainerStarted","Data":"248529f6ff62b0bc907a99a45700f345308e1f0adf3b56d1be283e2537091dcf"} Jul 05 19:35:52.627241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.627194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vsqt9" event={"ID":"c7441598-99c7-4b22-b6ac-5e956b534700","Type":"ContainerStarted","Data":"b3b93b9fb44254a512eeab5875387d31961b74e746a8478c15e7485ce59350e3"} Jul 05 19:35:52.644339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:35:52.644287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vsqt9" podStartSLOduration=1.644274196 podStartE2EDuration="1.644274196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:35:52.642935828 +0000 UTC m=+353.098719260" watchObservedRunningTime="2026-07-05 19:35:52.644274196 +0000 UTC m=+353.100057626" Jul 05 19:36:00.742937 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.742899 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh"] Jul 05 19:36:00.746445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.746428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.749262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.749239 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 05 19:36:00.749262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.749260 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 05 19:36:00.749415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.749335 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-txjr4\"" Jul 05 19:36:00.754211 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.754190 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh"] Jul 05 19:36:00.817411 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.817380 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.817573 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.817425 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.817573 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.817444 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hr4\" (UniqueName: \"kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.917829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.917803 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.917829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.917833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hr4\" (UniqueName: \"kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.918070 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.917891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.918227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.918208 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.918301 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.918225 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:00.929747 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:00.929706 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hr4\" (UniqueName: \"kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:01.056797 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:01.056726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:01.191142 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:01.191116 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh"] Jul 05 19:36:01.193300 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:01.193264 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b7546c_b200_4814_8e2e_89108a9d88ba.slice/crio-37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4 WatchSource:0}: Error finding container 37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4: Status 404 returned error can't find the container with id 37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4 Jul 05 19:36:01.668334 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:01.668299 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerID="4b4e6b385b4b5df703bdf877f763c611d91bdd238a6dce45ee34b966ecd29ef6" exitCode=0 Jul 05 19:36:01.668490 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:01.668374 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" event={"ID":"c9b7546c-b200-4814-8e2e-89108a9d88ba","Type":"ContainerDied","Data":"4b4e6b385b4b5df703bdf877f763c611d91bdd238a6dce45ee34b966ecd29ef6"} Jul 05 19:36:01.668490 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:01.668422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" event={"ID":"c9b7546c-b200-4814-8e2e-89108a9d88ba","Type":"ContainerStarted","Data":"37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4"} Jul 05 19:36:02.674041 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:02.674012 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerID="5c328f17297125b033562b811ece3c5ebc54464645898b2e9accd7d2a1c3fd87" exitCode=0 Jul 05 19:36:02.674429 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:02.674082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" event={"ID":"c9b7546c-b200-4814-8e2e-89108a9d88ba","Type":"ContainerDied","Data":"5c328f17297125b033562b811ece3c5ebc54464645898b2e9accd7d2a1c3fd87"} Jul 05 19:36:03.680507 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:03.680469 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerID="f38831281aa3ad932929fdf214b845a57deaa13dde91a32dc3ebde354c93e60d" exitCode=0 Jul 05 19:36:03.680895 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:03.680529 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" event={"ID":"c9b7546c-b200-4814-8e2e-89108a9d88ba","Type":"ContainerDied","Data":"f38831281aa3ad932929fdf214b845a57deaa13dde91a32dc3ebde354c93e60d"} Jul 05 19:36:04.809794 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.809772 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:04.851561 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.851538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8hr4\" (UniqueName: \"kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4\") pod \"c9b7546c-b200-4814-8e2e-89108a9d88ba\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " Jul 05 19:36:04.851661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.851575 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util\") pod \"c9b7546c-b200-4814-8e2e-89108a9d88ba\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " Jul 05 19:36:04.851661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.851594 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle\") pod \"c9b7546c-b200-4814-8e2e-89108a9d88ba\" (UID: \"c9b7546c-b200-4814-8e2e-89108a9d88ba\") " Jul 05 19:36:04.852314 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.852290 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle" (OuterVolumeSpecName: "bundle") pod "c9b7546c-b200-4814-8e2e-89108a9d88ba" (UID: "c9b7546c-b200-4814-8e2e-89108a9d88ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:36:04.853675 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.853650 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4" (OuterVolumeSpecName: "kube-api-access-d8hr4") pod "c9b7546c-b200-4814-8e2e-89108a9d88ba" (UID: "c9b7546c-b200-4814-8e2e-89108a9d88ba"). InnerVolumeSpecName "kube-api-access-d8hr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:36:04.856951 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.856919 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util" (OuterVolumeSpecName: "util") pod "c9b7546c-b200-4814-8e2e-89108a9d88ba" (UID: "c9b7546c-b200-4814-8e2e-89108a9d88ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:36:04.953102 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.953012 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8hr4\" (UniqueName: \"kubernetes.io/projected/c9b7546c-b200-4814-8e2e-89108a9d88ba-kube-api-access-d8hr4\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:04.953102 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.953046 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:04.953102 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:04.953081 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9b7546c-b200-4814-8e2e-89108a9d88ba-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:05.690607 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:05.690580 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" Jul 05 19:36:05.690765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:05.690576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp8tkh" event={"ID":"c9b7546c-b200-4814-8e2e-89108a9d88ba","Type":"ContainerDied","Data":"37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4"} Jul 05 19:36:05.690765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:05.690708 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f47a169efefc2a02a81b638f28baa1d8b793ae1a704666020c3c6cc4af16b4" Jul 05 19:36:15.961133 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961091 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt"] Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961371 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="pull" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961381 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="pull" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961391 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="extract" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961396 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="extract" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961407 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="util" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961412 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="util" Jul 05 19:36:15.961498 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.961470 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b7546c-b200-4814-8e2e-89108a9d88ba" containerName="extract" Jul 05 19:36:15.964554 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.964539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:15.967967 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.967947 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k85hm\"" Jul 05 19:36:15.968096 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.967981 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 05 19:36:15.968418 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.968398 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 05 19:36:15.968637 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.968621 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 05 19:36:15.968728 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.968688 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 05 19:36:15.996319 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:15.996296 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt"] Jul 05 19:36:16.040549 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.040528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.040677 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.040562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.040677 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.040581 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgpvx\" (UniqueName: \"kubernetes.io/projected/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-kube-api-access-lgpvx\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.040677 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.040607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.040790 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.040701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141127 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141094 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141171 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgpvx\" (UniqueName: \"kubernetes.io/projected/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-kube-api-access-lgpvx\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141262 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.141652 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.141603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.143939 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.143915 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.144048 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.143915 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.144048 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.143983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.151010 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.150991 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgpvx\" (UniqueName: \"kubernetes.io/projected/2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6-kube-api-access-lgpvx\") pod \"opendatahub-operator-controller-manager-68674bcb-648kt\" (UID: \"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.275281 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.275208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:16.426346 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.426323 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt"] Jul 05 19:36:16.428472 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:16.428448 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2773d0a4_ce54_4ef6_ab53_aa2fb22b77d6.slice/crio-e86b9295e530f19da7aa41558c03353f2b247e66512eac4a2e65eafe0b867f44 WatchSource:0}: Error finding container e86b9295e530f19da7aa41558c03353f2b247e66512eac4a2e65eafe0b867f44: Status 404 returned error can't find the container with id e86b9295e530f19da7aa41558c03353f2b247e66512eac4a2e65eafe0b867f44 Jul 05 19:36:16.737020 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:16.736986 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" event={"ID":"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6","Type":"ContainerStarted","Data":"e86b9295e530f19da7aa41558c03353f2b247e66512eac4a2e65eafe0b867f44"} Jul 05 19:36:19.753347 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:19.753273 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" event={"ID":"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6","Type":"ContainerStarted","Data":"25b297f5564c16c1b51b7ecb3eb4b237fee0e5eb4c1d8d2cdef6467fbf38f33c"} Jul 05 19:36:20.764407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:20.764328 2581 generic.go:356] "Generic (PLEG): container finished" podID="2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6" containerID="25b297f5564c16c1b51b7ecb3eb4b237fee0e5eb4c1d8d2cdef6467fbf38f33c" exitCode=0 Jul 05 19:36:20.764758 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:20.764414 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" event={"ID":"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6","Type":"ContainerDied","Data":"25b297f5564c16c1b51b7ecb3eb4b237fee0e5eb4c1d8d2cdef6467fbf38f33c"} Jul 05 19:36:21.771423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:21.771393 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" event={"ID":"2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6","Type":"ContainerStarted","Data":"3fd342ac7305cf83917adf70ae315f13a88d39ad82a758c61b229692ff6245d7"} Jul 05 19:36:21.771790 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:21.771529 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:21.796801 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:21.796754 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" podStartSLOduration=2.447983502 podStartE2EDuration="6.796737124s" totalImagesPullingTime="3.32467965s" totalInitContainerRuntime="1.024073972s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:21.794073889 +0000 UTC m=+382.249857319" watchObservedRunningTime="2026-07-05 19:36:21.796737124 +0000 UTC m=+382.252520557" Jul 05 19:36:32.777407 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:32.777378 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-648kt" Jul 05 19:36:41.588835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.588799 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-5vhjg"] Jul 05 19:36:41.597796 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.597771 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.600711 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.600688 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 05 19:36:41.600823 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.600719 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w598p\"" Jul 05 19:36:41.601795 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.601776 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 05 19:36:41.601950 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.601830 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 05 19:36:41.601950 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.601888 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 05 19:36:41.604308 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.604285 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-5vhjg"] Jul 05 19:36:41.758314 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.758278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tls-certs\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.758468 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.758335 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tmp\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.758468 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.758406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjwm\" (UniqueName: \"kubernetes.io/projected/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-kube-api-access-jgjwm\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.769625 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.769599 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2kbrx"] Jul 05 19:36:41.772383 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.772369 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:41.775427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.775408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-79z5h\"" Jul 05 19:36:41.775525 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.775444 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 05 19:36:41.794996 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.794972 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2kbrx"] Jul 05 19:36:41.859938 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.859866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tmp\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.859938 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.859916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjwm\" (UniqueName: \"kubernetes.io/projected/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-kube-api-access-jgjwm\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.860172 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.859954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:41.860172 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.859995 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmzx\" (UniqueName: \"kubernetes.io/projected/cfb0bcc4-f937-4901-bba7-6d76871b3a73-kube-api-access-fnmzx\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:41.860172 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.860036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tls-certs\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.862643 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.862611 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tmp\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.862766 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.862710 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-tls-certs\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.873544 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.873516 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjwm\" (UniqueName: \"kubernetes.io/projected/ba0f13ae-ef6b-4aec-9489-9636efa3a0b6-kube-api-access-jgjwm\") pod \"kube-auth-proxy-7484578f88-5vhjg\" (UID: \"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.908689 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.908668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" Jul 05 19:36:41.960695 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.960616 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:41.960846 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:36:41.960777 2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 05 19:36:41.960912 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:36:41.960853 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert podName:cfb0bcc4-f937-4901-bba7-6d76871b3a73 nodeName:}" failed. No retries permitted until 2026-07-05 19:36:42.460830661 +0000 UTC m=+402.916614070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert") pod "odh-model-controller-7998c67dc4-2kbrx" (UID: "cfb0bcc4-f937-4901-bba7-6d76871b3a73") : secret "odh-model-controller-webhook-cert" not found Jul 05 19:36:41.961003 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.960979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmzx\" (UniqueName: \"kubernetes.io/projected/cfb0bcc4-f937-4901-bba7-6d76871b3a73-kube-api-access-fnmzx\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:41.973111 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:41.973087 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmzx\" (UniqueName: \"kubernetes.io/projected/cfb0bcc4-f937-4901-bba7-6d76871b3a73-kube-api-access-fnmzx\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:42.051580 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.051559 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-5vhjg"] Jul 05 19:36:42.053561 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:42.053534 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0f13ae_ef6b_4aec_9489_9636efa3a0b6.slice/crio-815fb5fec3f7b0d77cc929187cd8e978fcbec281bd2ac449e44351b95bb18fee WatchSource:0}: Error finding container 815fb5fec3f7b0d77cc929187cd8e978fcbec281bd2ac449e44351b95bb18fee: Status 404 returned error can't find the container with id 815fb5fec3f7b0d77cc929187cd8e978fcbec281bd2ac449e44351b95bb18fee Jul 05 19:36:42.466516 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.466486 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:42.468911 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.468886 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfb0bcc4-f937-4901-bba7-6d76871b3a73-cert\") pod \"odh-model-controller-7998c67dc4-2kbrx\" (UID: \"cfb0bcc4-f937-4901-bba7-6d76871b3a73\") " pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:42.563835 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.563803 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mmrr2"] Jul 05 19:36:42.567991 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.567975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.570618 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.570596 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 05 19:36:42.570716 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.570698 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gwbbz\"" Jul 05 19:36:42.587300 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.587281 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mmrr2"] Jul 05 19:36:42.668114 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.668083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpzc\" (UniqueName: \"kubernetes.io/projected/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-kube-api-access-2bpzc\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.668573 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.668145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-tls-certs\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.683125 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.683086 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:42.769305 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.769270 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-tls-certs\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.769458 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.769360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpzc\" (UniqueName: \"kubernetes.io/projected/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-kube-api-access-2bpzc\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.772232 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.772198 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-tls-certs\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.789993 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.789943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpzc\" (UniqueName: \"kubernetes.io/projected/8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed-kube-api-access-2bpzc\") pod \"model-serving-api-54d89bd5fc-mmrr2\" (UID: \"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed\") " pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:42.857175 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.857142 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2kbrx"] Jul 05 19:36:42.860651 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:42.860624 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb0bcc4_f937_4901_bba7_6d76871b3a73.slice/crio-dfc0cf0a76595697e9552e9739729886da28f5aa26618c56a79bfc3af9f69edb WatchSource:0}: Error finding container dfc0cf0a76595697e9552e9739729886da28f5aa26618c56a79bfc3af9f69edb: Status 404 returned error can't find the container with id dfc0cf0a76595697e9552e9739729886da28f5aa26618c56a79bfc3af9f69edb Jul 05 19:36:42.871083 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.871030 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" event={"ID":"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6","Type":"ContainerStarted","Data":"815fb5fec3f7b0d77cc929187cd8e978fcbec281bd2ac449e44351b95bb18fee"} Jul 05 19:36:42.872268 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.872244 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" event={"ID":"cfb0bcc4-f937-4901-bba7-6d76871b3a73","Type":"ContainerStarted","Data":"dfc0cf0a76595697e9552e9739729886da28f5aa26618c56a79bfc3af9f69edb"} Jul 05 19:36:42.878711 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:42.878494 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:43.032882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.032809 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-mmrr2"] Jul 05 19:36:43.038153 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.038127 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:36:43.050418 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.050397 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.053367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.053183 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 05 19:36:43.053367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.053195 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 05 19:36:43.053367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.053215 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 05 19:36:43.053367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.053234 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 05 19:36:43.053565 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.053516 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8nc8c\"" Jul 05 19:36:43.058785 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.058763 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:36:43.173095 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173276 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173276 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tw47\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.173551 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.173455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.274720 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.274899 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274731 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.274899 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.274899 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.274899 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.275139 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274903 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.275139 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.274930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tw47\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.275919 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.275888 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.277635 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.277607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.277764 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.277696 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.278527 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.278500 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.278614 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.278527 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.285131 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.285077 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tw47\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.286850 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.286830 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token\") pod \"istiod-openshift-gateway-7574459656-f5bjg\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.361781 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.361735 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:43.532682 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.532650 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:36:43.535527 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:43.535467 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a6dc9_fbec_49fd_9263_f87a0c948afc.slice/crio-566ed5d9dfc454ad911e9183b7f99d82441895e5547f960c0763b5cbae1aa20a WatchSource:0}: Error finding container 566ed5d9dfc454ad911e9183b7f99d82441895e5547f960c0763b5cbae1aa20a: Status 404 returned error can't find the container with id 566ed5d9dfc454ad911e9183b7f99d82441895e5547f960c0763b5cbae1aa20a Jul 05 19:36:43.880116 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.879968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" event={"ID":"4f8a6dc9-fbec-49fd-9263-f87a0c948afc","Type":"ContainerStarted","Data":"566ed5d9dfc454ad911e9183b7f99d82441895e5547f960c0763b5cbae1aa20a"} Jul 05 19:36:43.882706 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:43.882565 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" event={"ID":"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed","Type":"ContainerStarted","Data":"abccf443ea3a8e6596a1713eb2301b6f1865b4e54176623a0c408a21d65aed52"} Jul 05 19:36:46.901628 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.901591 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" event={"ID":"cfb0bcc4-f937-4901-bba7-6d76871b3a73","Type":"ContainerStarted","Data":"49f9cab305544ef3bd6973a42f07321ffc7d9f98ebd090fdc6b9f4824cd40b82"} Jul 05 19:36:46.902091 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.901751 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:46.903392 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.903368 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" event={"ID":"ba0f13ae-ef6b-4aec-9489-9636efa3a0b6","Type":"ContainerStarted","Data":"2019ce13d377a6cef60b3735371a8563ed94316c0a798308053627f25e7ce425"} Jul 05 19:36:46.904938 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.904900 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" event={"ID":"8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed","Type":"ContainerStarted","Data":"6474cd7fd08adf69ce9145fc2a532c2f95ccc3494aeb7f65be6c5472fac412b7"} Jul 05 19:36:46.905045 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.905036 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:46.918134 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.918096 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" podStartSLOduration=2.063316922 podStartE2EDuration="5.918080366s" totalImagesPullingTime="3.854763444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:46.917799486 +0000 UTC m=+407.373582918" watchObservedRunningTime="2026-07-05 19:36:46.918080366 +0000 UTC m=+407.373863786" Jul 05 19:36:46.935714 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.935627 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" podStartSLOduration=1.268187412 podStartE2EDuration="4.935614318s" totalImagesPullingTime="3.667426906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:46.934379862 +0000 UTC m=+407.390163291" watchObservedRunningTime="2026-07-05 19:36:46.935614318 +0000 UTC m=+407.391397737" Jul 05 19:36:46.957792 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:46.957736 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7484578f88-5vhjg" podStartSLOduration=1.3087178019999999 podStartE2EDuration="5.957709401s" totalImagesPullingTime="4.648991599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:46.956177895 +0000 UTC m=+407.411961326" watchObservedRunningTime="2026-07-05 19:36:46.957709401 +0000 UTC m=+407.413492828" Jul 05 19:36:48.290431 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:48.290391 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 05 19:36:48.290717 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:48.290460 2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 05 19:36:48.917112 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:48.917082 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" event={"ID":"4f8a6dc9-fbec-49fd-9263-f87a0c948afc","Type":"ContainerStarted","Data":"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54"} Jul 05 19:36:48.917302 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:48.917213 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:48.957177 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:48.957129 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" podStartSLOduration=1.205629368 podStartE2EDuration="5.957113741s" totalImagesPullingTime="4.751484373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:48.954626612 +0000 UTC m=+409.410410040" watchObservedRunningTime="2026-07-05 19:36:48.957113741 +0000 UTC m=+409.412897171" Jul 05 19:36:49.923633 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:49.923603 2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-f5bjg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 05 19:36:49.923989 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:49.923657 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 19:36:52.923371 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:52.923344 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:53.213388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.213306 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp"] Jul 05 19:36:53.216774 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.216753 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.220521 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.220500 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-798gw\"" Jul 05 19:36:53.221521 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.221505 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 05 19:36:53.231253 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.231231 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c"] Jul 05 19:36:53.234733 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.234607 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.237496 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.237476 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 05 19:36:53.240582 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.240563 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bzxzs\"" Jul 05 19:36:53.245814 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.245795 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp"] Jul 05 19:36:53.250758 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.250740 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c"] Jul 05 19:36:53.273273 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.273243 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5a1198-388f-42dd-b505-59ffc6c818d5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.273415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.273311 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9963cb83-7685-498e-a64d-5a48e4328664-cert\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.273415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.273391 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk256\" (UniqueName: \"kubernetes.io/projected/2b5a1198-388f-42dd-b505-59ffc6c818d5-kube-api-access-gk256\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.273499 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.273419 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpjd\" (UniqueName: \"kubernetes.io/projected/9963cb83-7685-498e-a64d-5a48e4328664-kube-api-access-fdpjd\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.374030 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.373997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9963cb83-7685-498e-a64d-5a48e4328664-cert\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.374240 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.374046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk256\" (UniqueName: \"kubernetes.io/projected/2b5a1198-388f-42dd-b505-59ffc6c818d5-kube-api-access-gk256\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.374240 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.374088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpjd\" (UniqueName: \"kubernetes.io/projected/9963cb83-7685-498e-a64d-5a48e4328664-kube-api-access-fdpjd\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.374240 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.374108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5a1198-388f-42dd-b505-59ffc6c818d5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.376684 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.376659 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b5a1198-388f-42dd-b505-59ffc6c818d5-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.376812 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.376755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9963cb83-7685-498e-a64d-5a48e4328664-cert\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.389196 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.389168 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk256\" (UniqueName: \"kubernetes.io/projected/2b5a1198-388f-42dd-b505-59ffc6c818d5-kube-api-access-gk256\") pod \"llmisvc-controller-manager-777c6bd7d5-rzk4c\" (UID: \"2b5a1198-388f-42dd-b505-59ffc6c818d5\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.392905 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.392877 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpjd\" (UniqueName: \"kubernetes.io/projected/9963cb83-7685-498e-a64d-5a48e4328664-kube-api-access-fdpjd\") pod \"kserve-controller-manager-d6f4b65b8-lfkxp\" (UID: \"9963cb83-7685-498e-a64d-5a48e4328664\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.527002 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.526929 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:53.544936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.544915 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:53.670250 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.670224 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp"] Jul 05 19:36:53.672963 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:53.672934 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9963cb83_7685_498e_a64d_5a48e4328664.slice/crio-9e0844938371d411545dc2386918688373fc0d8848670b807b0f86dd22e455ce WatchSource:0}: Error finding container 9e0844938371d411545dc2386918688373fc0d8848670b807b0f86dd22e455ce: Status 404 returned error can't find the container with id 9e0844938371d411545dc2386918688373fc0d8848670b807b0f86dd22e455ce Jul 05 19:36:53.689003 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.688983 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c"] Jul 05 19:36:53.690527 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:36:53.690505 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b5a1198_388f_42dd_b505_59ffc6c818d5.slice/crio-50c3f54e983c2bdc1d1ba64c07bbb6cdb84fb2cdcfe9653cbfff7a3a33faf1aa WatchSource:0}: Error finding container 50c3f54e983c2bdc1d1ba64c07bbb6cdb84fb2cdcfe9653cbfff7a3a33faf1aa: Status 404 returned error can't find the container with id 50c3f54e983c2bdc1d1ba64c07bbb6cdb84fb2cdcfe9653cbfff7a3a33faf1aa Jul 05 19:36:53.943458 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.943422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" event={"ID":"9963cb83-7685-498e-a64d-5a48e4328664","Type":"ContainerStarted","Data":"9e0844938371d411545dc2386918688373fc0d8848670b807b0f86dd22e455ce"} Jul 05 19:36:53.944571 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:53.944546 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" event={"ID":"2b5a1198-388f-42dd-b505-59ffc6c818d5","Type":"ContainerStarted","Data":"50c3f54e983c2bdc1d1ba64c07bbb6cdb84fb2cdcfe9653cbfff7a3a33faf1aa"} Jul 05 19:36:56.959224 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.959188 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" event={"ID":"2b5a1198-388f-42dd-b505-59ffc6c818d5","Type":"ContainerStarted","Data":"03ef4a7408bed8cd1a3b6a4f114d20a940e4e0c33890bd9ebcdaf77027f4e839"} Jul 05 19:36:56.959620 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.959266 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:36:56.960725 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.960705 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" event={"ID":"9963cb83-7685-498e-a64d-5a48e4328664","Type":"ContainerStarted","Data":"5c0001c25b2d94b3efe1affddc7c691be64248b5978f13b3d19567256e73380b"} Jul 05 19:36:56.960849 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.960836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:36:56.977593 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.977553 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" podStartSLOduration=1.0529495739999999 podStartE2EDuration="3.977538237s" totalImagesPullingTime="2.924588663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:56.976468553 +0000 UTC m=+417.432251983" watchObservedRunningTime="2026-07-05 19:36:56.977538237 +0000 UTC m=+417.433321665" Jul 05 19:36:56.994456 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:56.994414 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" podStartSLOduration=1.699442 podStartE2EDuration="3.994399323s" totalImagesPullingTime="2.294957323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:56.993133096 +0000 UTC m=+417.448916523" watchObservedRunningTime="2026-07-05 19:36:56.994399323 +0000 UTC m=+417.450182751" Jul 05 19:36:57.913290 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:57.913260 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2kbrx" Jul 05 19:36:57.915306 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:57.915285 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-mmrr2" Jul 05 19:36:58.539475 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.539443 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:36:58.539849 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.539633 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerName="discovery" containerID="cri-o://ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54" gracePeriod=30 Jul 05 19:36:58.786361 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.786338 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:58.823404 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823331 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823404 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823377 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823434 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823475 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823523 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823583 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823549 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tw47\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.823765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823599 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert\") pod \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\" (UID: \"4f8a6dc9-fbec-49fd-9263-f87a0c948afc\") " Jul 05 19:36:58.824118 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.823978 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:36:58.826338 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.826304 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs" (OuterVolumeSpecName: "local-certs") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:36:58.826555 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.826529 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts" (OuterVolumeSpecName: "cacerts") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:36:58.826632 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.826555 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:36:58.826632 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.826578 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47" (OuterVolumeSpecName: "kube-api-access-2tw47") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "kube-api-access-2tw47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:36:58.827394 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.827376 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:36:58.827555 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.827539 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4f8a6dc9-fbec-49fd-9263-f87a0c948afc" (UID: "4f8a6dc9-fbec-49fd-9263-f87a0c948afc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:36:58.924623 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924591 2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-token\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924623 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924615 2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-local-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924623 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924623 2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-cacerts\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924623 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924630 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-ca-configmap\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924639 2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-kubeconfig\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924646 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tw47\" (UniqueName: \"kubernetes.io/projected/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-kube-api-access-2tw47\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.924852 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.924654 2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f8a6dc9-fbec-49fd-9263-f87a0c948afc-istio-csr-dns-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:36:58.972419 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.972382 2581 generic.go:356] "Generic (PLEG): container finished" podID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerID="ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54" exitCode=0 Jul 05 19:36:58.972562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.972433 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" Jul 05 19:36:58.972562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.972465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" event={"ID":"4f8a6dc9-fbec-49fd-9263-f87a0c948afc","Type":"ContainerDied","Data":"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54"} Jul 05 19:36:58.972562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.972519 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg" event={"ID":"4f8a6dc9-fbec-49fd-9263-f87a0c948afc","Type":"ContainerDied","Data":"566ed5d9dfc454ad911e9183b7f99d82441895e5547f960c0763b5cbae1aa20a"} Jul 05 19:36:58.972562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.972538 2581 scope.go:117] "RemoveContainer" containerID="ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54" Jul 05 19:36:58.981424 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.981411 2581 scope.go:117] "RemoveContainer" containerID="ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54" Jul 05 19:36:58.981678 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:36:58.981659 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54\": container with ID starting with ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54 not found: ID does not exist" containerID="ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54" Jul 05 19:36:58.987100 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.981823 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54"} err="failed to get container status \"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54\": rpc error: code = NotFound desc = could not find container \"ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54\": container with ID starting with ce0d903b9ae618306ef5c9345a9e44405ef369639762d5e7bdcbd6fd094d4d54 not found: ID does not exist" Jul 05 19:36:58.997215 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:58.997189 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:36:59.001222 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:36:59.001200 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f5bjg"] Jul 05 19:37:00.061429 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:00.061396 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" path="/var/lib/kubelet/pods/4f8a6dc9-fbec-49fd-9263-f87a0c948afc/volumes" Jul 05 19:37:27.968495 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:27.968419 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rzk4c" Jul 05 19:37:27.971297 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:27.971281 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lfkxp" Jul 05 19:37:37.427514 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.427480 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:37.428105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.428039 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerName="discovery" Jul 05 19:37:37.428105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.428080 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerName="discovery" Jul 05 19:37:37.428233 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.428161 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f8a6dc9-fbec-49fd-9263-f87a0c948afc" containerName="discovery" Jul 05 19:37:37.431288 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.431267 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:37.434318 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.434279 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4nfdj\"" Jul 05 19:37:37.435588 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.435566 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 05 19:37:37.435702 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.435566 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 05 19:37:37.439005 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.438852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x\") pod \"kuadrant-operator-catalog-tmqk9\" (UID: \"746816d7-e9e2-4d00-a8b5-ec16cb2666d9\") " pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:37.446776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.445635 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:37.540040 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.539998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x\") pod \"kuadrant-operator-catalog-tmqk9\" (UID: \"746816d7-e9e2-4d00-a8b5-ec16cb2666d9\") " pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:37.548476 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.548441 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x\") pod \"kuadrant-operator-catalog-tmqk9\" (UID: \"746816d7-e9e2-4d00-a8b5-ec16cb2666d9\") " pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:37.743969 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.743889 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:37.799468 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.799437 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:37.876288 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:37.876264 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:37.878546 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:37.878523 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746816d7_e9e2_4d00_a8b5_ec16cb2666d9.slice/crio-66ce56520e1bc8990aa4bf03c4d7bf4dc877ec5f0845ed6ede2612617da680e5 WatchSource:0}: Error finding container 66ce56520e1bc8990aa4bf03c4d7bf4dc877ec5f0845ed6ede2612617da680e5: Status 404 returned error can't find the container with id 66ce56520e1bc8990aa4bf03c4d7bf4dc877ec5f0845ed6ede2612617da680e5 Jul 05 19:37:38.008428 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.008355 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jmn84"] Jul 05 19:37:38.012759 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.012740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:38.020500 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.020467 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jmn84"] Jul 05 19:37:38.043117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.043090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcjv\" (UniqueName: \"kubernetes.io/projected/7f962fcb-6c4a-441a-844e-a8b57dfbd1d4-kube-api-access-7xcjv\") pod \"kuadrant-operator-catalog-jmn84\" (UID: \"7f962fcb-6c4a-441a-844e-a8b57dfbd1d4\") " pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:38.143859 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.143828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcjv\" (UniqueName: \"kubernetes.io/projected/7f962fcb-6c4a-441a-844e-a8b57dfbd1d4-kube-api-access-7xcjv\") pod \"kuadrant-operator-catalog-jmn84\" (UID: \"7f962fcb-6c4a-441a-844e-a8b57dfbd1d4\") " pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:38.144857 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.144829 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" event={"ID":"746816d7-e9e2-4d00-a8b5-ec16cb2666d9","Type":"ContainerStarted","Data":"66ce56520e1bc8990aa4bf03c4d7bf4dc877ec5f0845ed6ede2612617da680e5"} Jul 05 19:37:38.153541 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.153514 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcjv\" (UniqueName: \"kubernetes.io/projected/7f962fcb-6c4a-441a-844e-a8b57dfbd1d4-kube-api-access-7xcjv\") pod \"kuadrant-operator-catalog-jmn84\" (UID: \"7f962fcb-6c4a-441a-844e-a8b57dfbd1d4\") " pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:38.323415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.323324 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:38.471193 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:38.471165 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jmn84"] Jul 05 19:37:38.505816 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:38.505779 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f962fcb_6c4a_441a_844e_a8b57dfbd1d4.slice/crio-da945f302f0d6eb2f04e47cdf31e807bfb6fdcae8ac28d43365701773335d700 WatchSource:0}: Error finding container da945f302f0d6eb2f04e47cdf31e807bfb6fdcae8ac28d43365701773335d700: Status 404 returned error can't find the container with id da945f302f0d6eb2f04e47cdf31e807bfb6fdcae8ac28d43365701773335d700 Jul 05 19:37:39.152580 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:39.152522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" event={"ID":"7f962fcb-6c4a-441a-844e-a8b57dfbd1d4","Type":"ContainerStarted","Data":"da945f302f0d6eb2f04e47cdf31e807bfb6fdcae8ac28d43365701773335d700"} Jul 05 19:37:40.159573 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.159537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" event={"ID":"746816d7-e9e2-4d00-a8b5-ec16cb2666d9","Type":"ContainerStarted","Data":"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a"} Jul 05 19:37:40.160014 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.159593 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" podUID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" containerName="registry-server" containerID="cri-o://ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a" gracePeriod=2 Jul 05 19:37:40.162148 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.162118 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" event={"ID":"7f962fcb-6c4a-441a-844e-a8b57dfbd1d4","Type":"ContainerStarted","Data":"a85614f322ed7426530482d470d5615d8fdb460d775b174ce73ad1ae84f983d9"} Jul 05 19:37:40.177415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.177237 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" podStartSLOduration=1.037312307 podStartE2EDuration="3.177218506s" totalImagesPullingTime="2.139906199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:37:40.175102041 +0000 UTC m=+460.630885473" watchObservedRunningTime="2026-07-05 19:37:40.177218506 +0000 UTC m=+460.633001942" Jul 05 19:37:40.193342 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.193291 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" podStartSLOduration=1.677853642 podStartE2EDuration="3.193264483s" totalImagesPullingTime="1.515410841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:37:40.191278487 +0000 UTC m=+460.647061918" watchObservedRunningTime="2026-07-05 19:37:40.193264483 +0000 UTC m=+460.649047912" Jul 05 19:37:40.506915 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.506893 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:40.561043 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.560968 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x\") pod \"746816d7-e9e2-4d00-a8b5-ec16cb2666d9\" (UID: \"746816d7-e9e2-4d00-a8b5-ec16cb2666d9\") " Jul 05 19:37:40.563236 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.563211 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x" (OuterVolumeSpecName: "kube-api-access-zfn6x") pod "746816d7-e9e2-4d00-a8b5-ec16cb2666d9" (UID: "746816d7-e9e2-4d00-a8b5-ec16cb2666d9"). InnerVolumeSpecName "kube-api-access-zfn6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:37:40.661454 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:40.661424 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfn6x\" (UniqueName: \"kubernetes.io/projected/746816d7-e9e2-4d00-a8b5-ec16cb2666d9-kube-api-access-zfn6x\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:41.167291 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.167255 2581 generic.go:356] "Generic (PLEG): container finished" podID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" containerID="ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a" exitCode=0 Jul 05 19:37:41.167645 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.167296 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" Jul 05 19:37:41.167645 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.167339 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" event={"ID":"746816d7-e9e2-4d00-a8b5-ec16cb2666d9","Type":"ContainerDied","Data":"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a"} Jul 05 19:37:41.167645 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.167384 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tmqk9" event={"ID":"746816d7-e9e2-4d00-a8b5-ec16cb2666d9","Type":"ContainerDied","Data":"66ce56520e1bc8990aa4bf03c4d7bf4dc877ec5f0845ed6ede2612617da680e5"} Jul 05 19:37:41.167645 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.167404 2581 scope.go:117] "RemoveContainer" containerID="ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a" Jul 05 19:37:41.176867 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.176846 2581 scope.go:117] "RemoveContainer" containerID="ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a" Jul 05 19:37:41.177113 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:37:41.177090 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a\": container with ID starting with ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a not found: ID does not exist" containerID="ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a" Jul 05 19:37:41.177204 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.177118 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a"} err="failed to get container status \"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a\": rpc error: code = NotFound desc = could not find container \"ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a\": container with ID starting with ada7e8cbd440c125853da2751daa250d90156b5cb39f04095717f92013fe7c0a not found: ID does not exist" Jul 05 19:37:41.189491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.189468 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:41.193415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:41.193394 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tmqk9"] Jul 05 19:37:41.207677 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:37:41.207652 2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746816d7_e9e2_4d00_a8b5_ec16cb2666d9.slice\": RecentStats: unable to find data in memory cache]" Jul 05 19:37:42.065172 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:42.065140 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" path="/var/lib/kubelet/pods/746816d7-e9e2-4d00-a8b5-ec16cb2666d9/volumes" Jul 05 19:37:48.324305 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:48.324256 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:48.324785 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:48.324331 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:48.346815 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:48.346787 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:49.226584 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:49.226557 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jmn84" Jul 05 19:37:53.047819 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.047788 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj"] Jul 05 19:37:53.048322 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.048303 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" containerName="registry-server" Jul 05 19:37:53.048392 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.048327 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" containerName="registry-server" Jul 05 19:37:53.048444 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.048432 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="746816d7-e9e2-4d00-a8b5-ec16cb2666d9" containerName="registry-server" Jul 05 19:37:53.051621 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.051603 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.054636 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.054620 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wffnq\"" Jul 05 19:37:53.061159 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.061136 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj"] Jul 05 19:37:53.172247 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.172210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt24l\" (UniqueName: \"kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.172408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.172263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.172408 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.172294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.273450 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.273420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt24l\" (UniqueName: \"kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.273619 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.273471 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.273619 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.273518 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.273934 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.273918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.273978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.273947 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.283277 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.283254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt24l\" (UniqueName: \"kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.360624 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.360545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:53.447647 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.447617 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv"] Jul 05 19:37:53.453255 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.453233 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.459943 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.459921 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv"] Jul 05 19:37:53.500102 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.500081 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj"] Jul 05 19:37:53.501468 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:53.501440 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda531734b_ba20_4e1f_9ba2_ee74704d1be3.slice/crio-06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a WatchSource:0}: Error finding container 06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a: Status 404 returned error can't find the container with id 06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a Jul 05 19:37:53.578359 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.578324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.578359 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.578358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.578542 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.578486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr79h\" (UniqueName: \"kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.679388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.679362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr79h\" (UniqueName: \"kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.679508 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.679401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.679508 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.679432 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.679759 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.679738 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.679818 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.679771 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.688113 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.688094 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr79h\" (UniqueName: \"kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.765598 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.765573 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:53.844260 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.844230 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr"] Jul 05 19:37:53.849480 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.849453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:53.856567 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.856504 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr"] Jul 05 19:37:53.900928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.900903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv"] Jul 05 19:37:53.902880 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:53.902853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afeb7a_3561_4e31_af88_7acb57ba96de.slice/crio-dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68 WatchSource:0}: Error finding container dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68: Status 404 returned error can't find the container with id dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68 Jul 05 19:37:53.982654 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.982623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:53.982789 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.982669 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:53.982789 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:53.982705 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75nx9\" (UniqueName: \"kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.083981 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.083951 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.084315 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.083990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.084315 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.084024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75nx9\" (UniqueName: \"kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.084395 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.084331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.084395 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.084350 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.093415 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.093385 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75nx9\" (UniqueName: \"kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.161173 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.161151 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:54.232807 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.232776 2581 generic.go:356] "Generic (PLEG): container finished" podID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerID="90b781075cd10e5750eace076c3cd394405ef400da3aee3974451b6b4c940d99" exitCode=0 Jul 05 19:37:54.232932 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.232861 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" event={"ID":"67afeb7a-3561-4e31-af88-7acb57ba96de","Type":"ContainerDied","Data":"90b781075cd10e5750eace076c3cd394405ef400da3aee3974451b6b4c940d99"} Jul 05 19:37:54.232932 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.232918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" event={"ID":"67afeb7a-3561-4e31-af88-7acb57ba96de","Type":"ContainerStarted","Data":"dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68"} Jul 05 19:37:54.234731 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.234706 2581 generic.go:356] "Generic (PLEG): container finished" podID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerID="a17642225f247695a131243ea34c2dcb8301dbe8f12892a9f9388b5a382e1eb1" exitCode=0 Jul 05 19:37:54.234824 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.234753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" event={"ID":"a531734b-ba20-4e1f-9ba2-ee74704d1be3","Type":"ContainerDied","Data":"a17642225f247695a131243ea34c2dcb8301dbe8f12892a9f9388b5a382e1eb1"} Jul 05 19:37:54.234824 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.234798 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" event={"ID":"a531734b-ba20-4e1f-9ba2-ee74704d1be3","Type":"ContainerStarted","Data":"06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a"} Jul 05 19:37:54.249191 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.249169 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m"] Jul 05 19:37:54.253992 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.253974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.264622 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.264601 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m"] Jul 05 19:37:54.299426 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.299346 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr"] Jul 05 19:37:54.301633 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:54.301614 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aea3cd2_e33d_4348_b61c_3854195745e3.slice/crio-9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613 WatchSource:0}: Error finding container 9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613: Status 404 returned error can't find the container with id 9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613 Jul 05 19:37:54.388002 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.387970 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.388002 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.388003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4mv\" (UniqueName: \"kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.388246 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.388169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.489507 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.489449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.489507 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.489484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.489507 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.489504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4mv\" (UniqueName: \"kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.489848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.489826 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.489917 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.489901 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.498202 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.498177 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4mv\" (UniqueName: \"kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.564266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.564245 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:54.694291 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:54.694269 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m"] Jul 05 19:37:54.695925 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:37:54.695901 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118799c7_f94c_4ad4_9e63_0762113682fb.slice/crio-2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d WatchSource:0}: Error finding container 2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d: Status 404 returned error can't find the container with id 2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d Jul 05 19:37:55.241316 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.241282 2581 generic.go:356] "Generic (PLEG): container finished" podID="118799c7-f94c-4ad4-9e63-0762113682fb" containerID="3b72989b89cb89ce19ba8250dc85d6485af000dd4971a8d1396353f4f96c9e93" exitCode=0 Jul 05 19:37:55.241689 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.241373 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" event={"ID":"118799c7-f94c-4ad4-9e63-0762113682fb","Type":"ContainerDied","Data":"3b72989b89cb89ce19ba8250dc85d6485af000dd4971a8d1396353f4f96c9e93"} Jul 05 19:37:55.241689 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.241425 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" event={"ID":"118799c7-f94c-4ad4-9e63-0762113682fb","Type":"ContainerStarted","Data":"2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d"} Jul 05 19:37:55.243334 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.243311 2581 generic.go:356] "Generic (PLEG): container finished" podID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerID="1c7f7da6d5732330bccb498dec098a1a25223a006dff4fc7d40623f62c37d0b9" exitCode=0 Jul 05 19:37:55.243422 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.243401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" event={"ID":"a531734b-ba20-4e1f-9ba2-ee74704d1be3","Type":"ContainerDied","Data":"1c7f7da6d5732330bccb498dec098a1a25223a006dff4fc7d40623f62c37d0b9"} Jul 05 19:37:55.245414 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.245395 2581 generic.go:356] "Generic (PLEG): container finished" podID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerID="9ca437f87893c39158ba199fc39db8fabcf48918fabc4f5b9ba701cf1074631b" exitCode=0 Jul 05 19:37:55.245500 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.245479 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" event={"ID":"67afeb7a-3561-4e31-af88-7acb57ba96de","Type":"ContainerDied","Data":"9ca437f87893c39158ba199fc39db8fabcf48918fabc4f5b9ba701cf1074631b"} Jul 05 19:37:55.247117 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.247092 2581 generic.go:356] "Generic (PLEG): container finished" podID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerID="992d5eb950c06960bab3eba09e9fd70669de03ad685240c4d7853b0280e928c2" exitCode=0 Jul 05 19:37:55.247223 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.247129 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" event={"ID":"7aea3cd2-e33d-4348-b61c-3854195745e3","Type":"ContainerDied","Data":"992d5eb950c06960bab3eba09e9fd70669de03ad685240c4d7853b0280e928c2"} Jul 05 19:37:55.247223 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:55.247158 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" event={"ID":"7aea3cd2-e33d-4348-b61c-3854195745e3","Type":"ContainerStarted","Data":"9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613"} Jul 05 19:37:56.254016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.253937 2581 generic.go:356] "Generic (PLEG): container finished" podID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerID="58ecd60199a1adb7e795d3b47b5e6c673c285b66b5128bb62ef0728fc8a530d4" exitCode=0 Jul 05 19:37:56.254376 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.254023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" event={"ID":"a531734b-ba20-4e1f-9ba2-ee74704d1be3","Type":"ContainerDied","Data":"58ecd60199a1adb7e795d3b47b5e6c673c285b66b5128bb62ef0728fc8a530d4"} Jul 05 19:37:56.256118 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.256094 2581 generic.go:356] "Generic (PLEG): container finished" podID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerID="2faa876bd5ef79a4ce34e3d7173573760fba5504448334ade4887268217a092c" exitCode=0 Jul 05 19:37:56.256222 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.256170 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" event={"ID":"67afeb7a-3561-4e31-af88-7acb57ba96de","Type":"ContainerDied","Data":"2faa876bd5ef79a4ce34e3d7173573760fba5504448334ade4887268217a092c"} Jul 05 19:37:56.257815 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.257793 2581 generic.go:356] "Generic (PLEG): container finished" podID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerID="00d160ed3f81095ecc0f88429eac69ebdb6ecc355f7de3e1b6a573c78c498296" exitCode=0 Jul 05 19:37:56.257920 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.257818 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" event={"ID":"7aea3cd2-e33d-4348-b61c-3854195745e3","Type":"ContainerDied","Data":"00d160ed3f81095ecc0f88429eac69ebdb6ecc355f7de3e1b6a573c78c498296"} Jul 05 19:37:56.259738 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.259716 2581 generic.go:356] "Generic (PLEG): container finished" podID="118799c7-f94c-4ad4-9e63-0762113682fb" containerID="07315404e5720d9ca30dd392860cf59617b82d203269967e9078d4f1c35f48cd" exitCode=0 Jul 05 19:37:56.259821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:56.259746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" event={"ID":"118799c7-f94c-4ad4-9e63-0762113682fb","Type":"ContainerDied","Data":"07315404e5720d9ca30dd392860cf59617b82d203269967e9078d4f1c35f48cd"} Jul 05 19:37:57.266765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.266732 2581 generic.go:356] "Generic (PLEG): container finished" podID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerID="88829d9c1cb85fa5bdd59edc20fc9513f1fff838f47c4982194b23ea2bde7d8b" exitCode=0 Jul 05 19:37:57.267213 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.266815 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" event={"ID":"7aea3cd2-e33d-4348-b61c-3854195745e3","Type":"ContainerDied","Data":"88829d9c1cb85fa5bdd59edc20fc9513f1fff838f47c4982194b23ea2bde7d8b"} Jul 05 19:37:57.268817 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.268797 2581 generic.go:356] "Generic (PLEG): container finished" podID="118799c7-f94c-4ad4-9e63-0762113682fb" containerID="db1be64044570a1016f507f4a90e932613af072bc3b7304419687c4392afa761" exitCode=0 Jul 05 19:37:57.268935 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.268880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" event={"ID":"118799c7-f94c-4ad4-9e63-0762113682fb","Type":"ContainerDied","Data":"db1be64044570a1016f507f4a90e932613af072bc3b7304419687c4392afa761"} Jul 05 19:37:57.398370 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.398348 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:57.431640 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.431619 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:57.514046 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514019 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util\") pod \"67afeb7a-3561-4e31-af88-7acb57ba96de\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " Jul 05 19:37:57.514190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514085 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle\") pod \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " Jul 05 19:37:57.514190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514114 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt24l\" (UniqueName: \"kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l\") pod \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " Jul 05 19:37:57.514190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514162 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle\") pod \"67afeb7a-3561-4e31-af88-7acb57ba96de\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " Jul 05 19:37:57.514190 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514181 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr79h\" (UniqueName: \"kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h\") pod \"67afeb7a-3561-4e31-af88-7acb57ba96de\" (UID: \"67afeb7a-3561-4e31-af88-7acb57ba96de\") " Jul 05 19:37:57.514365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514216 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util\") pod \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\" (UID: \"a531734b-ba20-4e1f-9ba2-ee74704d1be3\") " Jul 05 19:37:57.514604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514578 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle" (OuterVolumeSpecName: "bundle") pod "a531734b-ba20-4e1f-9ba2-ee74704d1be3" (UID: "a531734b-ba20-4e1f-9ba2-ee74704d1be3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:57.514845 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.514822 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle" (OuterVolumeSpecName: "bundle") pod "67afeb7a-3561-4e31-af88-7acb57ba96de" (UID: "67afeb7a-3561-4e31-af88-7acb57ba96de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:57.516994 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.516931 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h" (OuterVolumeSpecName: "kube-api-access-nr79h") pod "67afeb7a-3561-4e31-af88-7acb57ba96de" (UID: "67afeb7a-3561-4e31-af88-7acb57ba96de"). InnerVolumeSpecName "kube-api-access-nr79h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:37:57.516994 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.516946 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l" (OuterVolumeSpecName: "kube-api-access-lt24l") pod "a531734b-ba20-4e1f-9ba2-ee74704d1be3" (UID: "a531734b-ba20-4e1f-9ba2-ee74704d1be3"). InnerVolumeSpecName "kube-api-access-lt24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:37:57.520116 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.520090 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util" (OuterVolumeSpecName: "util") pod "67afeb7a-3561-4e31-af88-7acb57ba96de" (UID: "67afeb7a-3561-4e31-af88-7acb57ba96de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:57.522543 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.522520 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util" (OuterVolumeSpecName: "util") pod "a531734b-ba20-4e1f-9ba2-ee74704d1be3" (UID: "a531734b-ba20-4e1f-9ba2-ee74704d1be3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:57.614821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614798 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:57.614821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614819 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:57.614821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614827 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a531734b-ba20-4e1f-9ba2-ee74704d1be3-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:57.614978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614836 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt24l\" (UniqueName: \"kubernetes.io/projected/a531734b-ba20-4e1f-9ba2-ee74704d1be3-kube-api-access-lt24l\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:57.614978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614845 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67afeb7a-3561-4e31-af88-7acb57ba96de-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:57.614978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:57.614853 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr79h\" (UniqueName: \"kubernetes.io/projected/67afeb7a-3561-4e31-af88-7acb57ba96de-kube-api-access-nr79h\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.280218 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.280185 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" Jul 05 19:37:58.280593 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.280223 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj" event={"ID":"a531734b-ba20-4e1f-9ba2-ee74704d1be3","Type":"ContainerDied","Data":"06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a"} Jul 05 19:37:58.280593 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.280308 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f0b5f4ff0a6499e081e56fba5ddf538e0a22bcda9bf18630498bf84b85e59a" Jul 05 19:37:58.282244 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.282225 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" Jul 05 19:37:58.282378 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.282254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv" event={"ID":"67afeb7a-3561-4e31-af88-7acb57ba96de","Type":"ContainerDied","Data":"dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68"} Jul 05 19:37:58.282378 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.282297 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc7e1c04550a6b9e1f4f2fb7113ca593c44395ccc0960b9878017b3444056d68" Jul 05 19:37:58.419180 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.419159 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:58.450367 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.450342 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:58.522840 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.522805 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75nx9\" (UniqueName: \"kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9\") pod \"7aea3cd2-e33d-4348-b61c-3854195745e3\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " Jul 05 19:37:58.523105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.522909 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle\") pod \"7aea3cd2-e33d-4348-b61c-3854195745e3\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " Jul 05 19:37:58.523105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.522935 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle\") pod \"118799c7-f94c-4ad4-9e63-0762113682fb\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " Jul 05 19:37:58.523105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.522962 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util\") pod \"118799c7-f94c-4ad4-9e63-0762113682fb\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " Jul 05 19:37:58.523105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.522991 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util\") pod \"7aea3cd2-e33d-4348-b61c-3854195745e3\" (UID: \"7aea3cd2-e33d-4348-b61c-3854195745e3\") " Jul 05 19:37:58.523105 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.523023 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt4mv\" (UniqueName: \"kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv\") pod \"118799c7-f94c-4ad4-9e63-0762113682fb\" (UID: \"118799c7-f94c-4ad4-9e63-0762113682fb\") " Jul 05 19:37:58.523618 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.523567 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle" (OuterVolumeSpecName: "bundle") pod "118799c7-f94c-4ad4-9e63-0762113682fb" (UID: "118799c7-f94c-4ad4-9e63-0762113682fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:58.523804 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.523587 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle" (OuterVolumeSpecName: "bundle") pod "7aea3cd2-e33d-4348-b61c-3854195745e3" (UID: "7aea3cd2-e33d-4348-b61c-3854195745e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:58.525385 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.525360 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9" (OuterVolumeSpecName: "kube-api-access-75nx9") pod "7aea3cd2-e33d-4348-b61c-3854195745e3" (UID: "7aea3cd2-e33d-4348-b61c-3854195745e3"). InnerVolumeSpecName "kube-api-access-75nx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:37:58.525489 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.525454 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv" (OuterVolumeSpecName: "kube-api-access-bt4mv") pod "118799c7-f94c-4ad4-9e63-0762113682fb" (UID: "118799c7-f94c-4ad4-9e63-0762113682fb"). InnerVolumeSpecName "kube-api-access-bt4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:37:58.532352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.532318 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util" (OuterVolumeSpecName: "util") pod "118799c7-f94c-4ad4-9e63-0762113682fb" (UID: "118799c7-f94c-4ad4-9e63-0762113682fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:58.532605 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.532588 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util" (OuterVolumeSpecName: "util") pod "7aea3cd2-e33d-4348-b61c-3854195745e3" (UID: "7aea3cd2-e33d-4348-b61c-3854195745e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624275 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624307 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt4mv\" (UniqueName: \"kubernetes.io/projected/118799c7-f94c-4ad4-9e63-0762113682fb-kube-api-access-bt4mv\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624320 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75nx9\" (UniqueName: \"kubernetes.io/projected/7aea3cd2-e33d-4348-b61c-3854195745e3-kube-api-access-75nx9\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624332 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7aea3cd2-e33d-4348-b61c-3854195745e3-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624343 2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:58.624365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:58.624353 2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/118799c7-f94c-4ad4-9e63-0762113682fb-util\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:37:59.288976 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.288944 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" Jul 05 19:37:59.288976 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.288960 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr" event={"ID":"7aea3cd2-e33d-4348-b61c-3854195745e3","Type":"ContainerDied","Data":"9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613"} Jul 05 19:37:59.289457 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.289011 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab86d85b64a282931d8a934c3d41eccb0b16fa84ae2d2a216ae03ff0cede613" Jul 05 19:37:59.290640 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.290621 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" Jul 05 19:37:59.290772 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.290619 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m" event={"ID":"118799c7-f94c-4ad4-9e63-0762113682fb","Type":"ContainerDied","Data":"2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d"} Jul 05 19:37:59.290772 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:37:59.290738 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dbb527b3fc79d1b1b7103e076719502597f32cace6a440add1e561ff9ab0a8d" Jul 05 19:38:00.295016 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.294925 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-675878ffb5-c9gx7"] Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295288 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295301 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295310 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295316 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295325 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295330 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295338 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295343 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295349 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295353 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295362 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295368 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295375 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295379 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295384 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295389 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295396 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295400 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="util" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295406 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295411 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295417 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295422 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295433 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295438 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="pull" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295483 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="118799c7-f94c-4ad4-9e63-0762113682fb" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295492 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aea3cd2-e33d-4348-b61c-3854195745e3" containerName="extract" Jul 05 19:38:00.295485 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295498 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a531734b-ba20-4e1f-9ba2-ee74704d1be3" containerName="extract" Jul 05 19:38:00.296713 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.295508 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67afeb7a-3561-4e31-af88-7acb57ba96de" containerName="extract" Jul 05 19:38:00.299863 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.299848 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.319772 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.319747 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675878ffb5-c9gx7"] Jul 05 19:38:00.437421 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nck\" (UniqueName: \"kubernetes.io/projected/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-kube-api-access-x5nck\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437421 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437442 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-oauth-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437674 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-service-ca\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437674 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437674 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437620 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437674 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-trusted-ca-bundle\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.437834 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.437681 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-oauth-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538613 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538582 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538625 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538655 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-trusted-ca-bundle\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538688 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-oauth-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nck\" (UniqueName: \"kubernetes.io/projected/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-kube-api-access-x5nck\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.538776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-oauth-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.539112 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.538789 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-service-ca\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.539553 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.539504 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.539661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.539579 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-service-ca\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.539661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.539586 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-oauth-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.539661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.539644 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-trusted-ca-bundle\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.541920 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.541897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-oauth-config\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.542006 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.541982 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-console-serving-cert\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.552119 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.551976 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nck\" (UniqueName: \"kubernetes.io/projected/bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3-kube-api-access-x5nck\") pod \"console-675878ffb5-c9gx7\" (UID: \"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3\") " pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.608882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.608825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:00.775734 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:00.775709 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-675878ffb5-c9gx7"] Jul 05 19:38:00.778027 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:00.778005 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf142d2_25f7_40b9_98ae_1d96fd4d9fc3.slice/crio-c4d498a08bbde5cc0c11465f2caf6eb39b4097a7cef8567e0d0cdb3c1f5e1c7d WatchSource:0}: Error finding container c4d498a08bbde5cc0c11465f2caf6eb39b4097a7cef8567e0d0cdb3c1f5e1c7d: Status 404 returned error can't find the container with id c4d498a08bbde5cc0c11465f2caf6eb39b4097a7cef8567e0d0cdb3c1f5e1c7d Jul 05 19:38:01.303419 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:01.303384 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675878ffb5-c9gx7" event={"ID":"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3","Type":"ContainerStarted","Data":"b4517cb7ad22f8b081fa514dfb518d7225f6814d08bf18be8d48e0d2d1119bb4"} Jul 05 19:38:01.303890 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:01.303440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-675878ffb5-c9gx7" event={"ID":"bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3","Type":"ContainerStarted","Data":"c4d498a08bbde5cc0c11465f2caf6eb39b4097a7cef8567e0d0cdb3c1f5e1c7d"} Jul 05 19:38:01.325275 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:01.325220 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-675878ffb5-c9gx7" podStartSLOduration=1.32520669 podStartE2EDuration="1.32520669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:01.322139283 +0000 UTC m=+481.777922714" watchObservedRunningTime="2026-07-05 19:38:01.32520669 +0000 UTC m=+481.780990120" Jul 05 19:38:03.814684 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.814646 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp"] Jul 05 19:38:03.818224 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.818206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:03.824438 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.824414 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8q5zs\"" Jul 05 19:38:03.824732 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.824710 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 05 19:38:03.835959 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.835930 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp"] Jul 05 19:38:03.968781 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:03.968743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq84g\" (UniqueName: \"kubernetes.io/projected/4259dc9b-4833-4c2e-99ef-f4579ed3ebb3-kube-api-access-tq84g\") pod \"dns-operator-controller-manager-864dd78c4c-8bqjp\" (UID: \"4259dc9b-4833-4c2e-99ef-f4579ed3ebb3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:04.070202 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:04.070122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq84g\" (UniqueName: \"kubernetes.io/projected/4259dc9b-4833-4c2e-99ef-f4579ed3ebb3-kube-api-access-tq84g\") pod \"dns-operator-controller-manager-864dd78c4c-8bqjp\" (UID: \"4259dc9b-4833-4c2e-99ef-f4579ed3ebb3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:04.081613 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:04.081584 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq84g\" (UniqueName: \"kubernetes.io/projected/4259dc9b-4833-4c2e-99ef-f4579ed3ebb3-kube-api-access-tq84g\") pod \"dns-operator-controller-manager-864dd78c4c-8bqjp\" (UID: \"4259dc9b-4833-4c2e-99ef-f4579ed3ebb3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:04.128691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:04.128649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:04.261946 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:04.261923 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp"] Jul 05 19:38:04.263729 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:04.263701 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259dc9b_4833_4c2e_99ef_f4579ed3ebb3.slice/crio-7c3ed9b22428c3fda17fb00c2d0b8ed11fa12c338779a8007b85df5219025bbb WatchSource:0}: Error finding container 7c3ed9b22428c3fda17fb00c2d0b8ed11fa12c338779a8007b85df5219025bbb: Status 404 returned error can't find the container with id 7c3ed9b22428c3fda17fb00c2d0b8ed11fa12c338779a8007b85df5219025bbb Jul 05 19:38:04.318240 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:04.318212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" event={"ID":"4259dc9b-4833-4c2e-99ef-f4579ed3ebb3","Type":"ContainerStarted","Data":"7c3ed9b22428c3fda17fb00c2d0b8ed11fa12c338779a8007b85df5219025bbb"} Jul 05 19:38:06.333474 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:06.333441 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" event={"ID":"4259dc9b-4833-4c2e-99ef-f4579ed3ebb3","Type":"ContainerStarted","Data":"869c23859493c265f17a6fa6e5fa337229f4ae48c46b92daef99ed9f8594fbe7"} Jul 05 19:38:06.333829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:06.333573 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:06.353990 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:06.353938 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" podStartSLOduration=1.470671986 podStartE2EDuration="3.353923206s" totalImagesPullingTime="1.88325122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:06.351469432 +0000 UTC m=+486.807252863" watchObservedRunningTime="2026-07-05 19:38:06.353923206 +0000 UTC m=+486.809706635" Jul 05 19:38:08.447974 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.447937 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8"] Jul 05 19:38:08.451448 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.451431 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:08.454414 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.454395 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lqdz6\"" Jul 05 19:38:08.460337 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.460313 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8"] Jul 05 19:38:08.613899 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.613868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kgv\" (UniqueName: \"kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv\") pod \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" (UID: \"1689be87-04f2-421c-8707-020ad45b02d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:08.714530 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.714431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kgv\" (UniqueName: \"kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv\") pod \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" (UID: \"1689be87-04f2-421c-8707-020ad45b02d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:08.723634 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.723606 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kgv\" (UniqueName: \"kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv\") pod \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" (UID: \"1689be87-04f2-421c-8707-020ad45b02d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:08.762536 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.762505 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:08.904347 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:08.904322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8"] Jul 05 19:38:08.905967 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:08.905941 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1689be87_04f2_421c_8707_020ad45b02d5.slice/crio-0d7e7f8d6b7ae0b6dcc0bcbdca92eec2669e1699b1b0a92dfba58f7a76c9a4e0 WatchSource:0}: Error finding container 0d7e7f8d6b7ae0b6dcc0bcbdca92eec2669e1699b1b0a92dfba58f7a76c9a4e0: Status 404 returned error can't find the container with id 0d7e7f8d6b7ae0b6dcc0bcbdca92eec2669e1699b1b0a92dfba58f7a76c9a4e0 Jul 05 19:38:09.349433 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:09.349402 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" event={"ID":"1689be87-04f2-421c-8707-020ad45b02d5","Type":"ContainerStarted","Data":"0d7e7f8d6b7ae0b6dcc0bcbdca92eec2669e1699b1b0a92dfba58f7a76c9a4e0"} Jul 05 19:38:10.609557 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:10.609534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:10.609924 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:10.609618 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:10.614121 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:10.614104 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:11.362014 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:11.361977 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" event={"ID":"1689be87-04f2-421c-8707-020ad45b02d5","Type":"ContainerStarted","Data":"cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd"} Jul 05 19:38:11.362207 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:11.362164 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:11.365821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:11.365802 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-675878ffb5-c9gx7" Jul 05 19:38:11.380601 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:11.380561 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" podStartSLOduration=1.7523194069999999 podStartE2EDuration="3.380546619s" totalImagesPullingTime="1.628227212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:11.378966375 +0000 UTC m=+491.834749806" watchObservedRunningTime="2026-07-05 19:38:11.380546619 +0000 UTC m=+491.836330048" Jul 05 19:38:11.426957 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:11.426932 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:38:12.808752 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:12.808713 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5"] Jul 05 19:38:12.814286 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:12.814259 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:12.817825 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:12.817800 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f44bw\"" Jul 05 19:38:12.817825 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:12.817822 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5"] Jul 05 19:38:12.951878 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:12.951842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjf4d\" (UniqueName: \"kubernetes.io/projected/bcf5f16a-94bb-4a20-b73a-39eb068ecdf2-kube-api-access-xjf4d\") pod \"authorino-operator-75cc8d5cb7-9xdp5\" (UID: \"bcf5f16a-94bb-4a20-b73a-39eb068ecdf2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:13.053219 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:13.053184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4d\" (UniqueName: \"kubernetes.io/projected/bcf5f16a-94bb-4a20-b73a-39eb068ecdf2-kube-api-access-xjf4d\") pod \"authorino-operator-75cc8d5cb7-9xdp5\" (UID: \"bcf5f16a-94bb-4a20-b73a-39eb068ecdf2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:13.062622 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:13.062550 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjf4d\" (UniqueName: \"kubernetes.io/projected/bcf5f16a-94bb-4a20-b73a-39eb068ecdf2-kube-api-access-xjf4d\") pod \"authorino-operator-75cc8d5cb7-9xdp5\" (UID: \"bcf5f16a-94bb-4a20-b73a-39eb068ecdf2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:13.125421 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:13.125393 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:13.257827 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:13.257798 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5"] Jul 05 19:38:13.259121 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:13.259092 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf5f16a_94bb_4a20_b73a_39eb068ecdf2.slice/crio-42e4ae0cc1fd58e73458b987fd9da0febb2b3e2067ee40ae90731ceafa3c2ff8 WatchSource:0}: Error finding container 42e4ae0cc1fd58e73458b987fd9da0febb2b3e2067ee40ae90731ceafa3c2ff8: Status 404 returned error can't find the container with id 42e4ae0cc1fd58e73458b987fd9da0febb2b3e2067ee40ae90731ceafa3c2ff8 Jul 05 19:38:13.373887 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:13.373817 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" event={"ID":"bcf5f16a-94bb-4a20-b73a-39eb068ecdf2","Type":"ContainerStarted","Data":"42e4ae0cc1fd58e73458b987fd9da0febb2b3e2067ee40ae90731ceafa3c2ff8"} Jul 05 19:38:16.391069 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:16.391023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" event={"ID":"bcf5f16a-94bb-4a20-b73a-39eb068ecdf2","Type":"ContainerStarted","Data":"4510d577cf4c818372c05aa65a504286908092892e05ca122597852353e97736"} Jul 05 19:38:16.391441 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:16.391106 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:16.411207 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:16.411165 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" podStartSLOduration=2.2499918660000002 podStartE2EDuration="4.411151615s" totalImagesPullingTime="2.161159749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:16.409042381 +0000 UTC m=+496.864825812" watchObservedRunningTime="2026-07-05 19:38:16.411151615 +0000 UTC m=+496.866935088" Jul 05 19:38:17.340345 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.340316 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8bqjp" Jul 05 19:38:17.787708 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.787676 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7"] Jul 05 19:38:17.791271 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.791256 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:17.796017 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.795994 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wk9wk\"" Jul 05 19:38:17.810524 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.810498 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7"] Jul 05 19:38:17.896392 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.896366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7rk\" (UniqueName: \"kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:17.896776 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.896411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:17.997049 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.997015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7rk\" (UniqueName: \"kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:17.997219 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.997085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:17.997390 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:17.997372 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:18.006837 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:18.006813 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7rk\" (UniqueName: \"kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:18.101139 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:18.100878 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:18.239328 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:18.239306 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7"] Jul 05 19:38:18.240969 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:18.240945 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a7b821_d433_4572_868b_1d736ad77056.slice/crio-b2383eccedb495deaa9d7a80747b2d32e7357a0f10eda3ba89ab09e72072cabc WatchSource:0}: Error finding container b2383eccedb495deaa9d7a80747b2d32e7357a0f10eda3ba89ab09e72072cabc: Status 404 returned error can't find the container with id b2383eccedb495deaa9d7a80747b2d32e7357a0f10eda3ba89ab09e72072cabc Jul 05 19:38:18.403092 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:18.403036 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" event={"ID":"b8a7b821-d433-4572-868b-1d736ad77056","Type":"ContainerStarted","Data":"b2383eccedb495deaa9d7a80747b2d32e7357a0f10eda3ba89ab09e72072cabc"} Jul 05 19:38:22.371347 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:22.371314 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:23.430999 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:23.430954 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" event={"ID":"b8a7b821-d433-4572-868b-1d736ad77056","Type":"ContainerStarted","Data":"691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade"} Jul 05 19:38:23.431406 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:23.431021 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:23.456482 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:23.456408 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" podStartSLOduration=1.486479876 podStartE2EDuration="6.456393556s" totalImagesPullingTime="4.96991368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:23.455882365 +0000 UTC m=+503.911665795" watchObservedRunningTime="2026-07-05 19:38:23.456393556 +0000 UTC m=+503.912176985" Jul 05 19:38:25.723892 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.723861 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8"] Jul 05 19:38:25.727624 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.727605 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.730942 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.730923 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wffnq\"" Jul 05 19:38:25.731534 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.731518 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 05 19:38:25.731721 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.731706 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 05 19:38:25.737170 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.737150 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8"] Jul 05 19:38:25.764859 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.764829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6xz\" (UniqueName: \"kubernetes.io/projected/efbfc0c7-6728-4e44-8e8e-4331657ccf06-kube-api-access-tk6xz\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.764859 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.764860 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efbfc0c7-6728-4e44-8e8e-4331657ccf06-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.765019 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.764884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.865972 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.865938 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6xz\" (UniqueName: \"kubernetes.io/projected/efbfc0c7-6728-4e44-8e8e-4331657ccf06-kube-api-access-tk6xz\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.866157 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.865977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efbfc0c7-6728-4e44-8e8e-4331657ccf06-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.866157 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.866002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.866157 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:25.866129 2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 05 19:38:25.866367 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:25.866200 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert podName:efbfc0c7-6728-4e44-8e8e-4331657ccf06 nodeName:}" failed. No retries permitted until 2026-07-05 19:38:26.366178083 +0000 UTC m=+506.821961503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-bbnj8" (UID: "efbfc0c7-6728-4e44-8e8e-4331657ccf06") : secret "plugin-serving-cert" not found Jul 05 19:38:25.866574 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.866555 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/efbfc0c7-6728-4e44-8e8e-4331657ccf06-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:25.875611 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:25.875590 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6xz\" (UniqueName: \"kubernetes.io/projected/efbfc0c7-6728-4e44-8e8e-4331657ccf06-kube-api-access-tk6xz\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:26.369386 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:26.369356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:26.371797 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:26.371770 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/efbfc0c7-6728-4e44-8e8e-4331657ccf06-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bbnj8\" (UID: \"efbfc0c7-6728-4e44-8e8e-4331657ccf06\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:26.636837 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:26.636761 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" Jul 05 19:38:26.787560 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:26.787539 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8"] Jul 05 19:38:26.789840 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:26.789819 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefbfc0c7_6728_4e44_8e8e_4331657ccf06.slice/crio-f13b52b1fe1ad1ee440120aac8c2121803cf6cf4a180081a4b8121dd9f59b5a6 WatchSource:0}: Error finding container f13b52b1fe1ad1ee440120aac8c2121803cf6cf4a180081a4b8121dd9f59b5a6: Status 404 returned error can't find the container with id f13b52b1fe1ad1ee440120aac8c2121803cf6cf4a180081a4b8121dd9f59b5a6 Jul 05 19:38:27.398767 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:27.398739 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9xdp5" Jul 05 19:38:27.451759 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:27.451727 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" event={"ID":"efbfc0c7-6728-4e44-8e8e-4331657ccf06","Type":"ContainerStarted","Data":"f13b52b1fe1ad1ee440120aac8c2121803cf6cf4a180081a4b8121dd9f59b5a6"} Jul 05 19:38:34.438817 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:34.438785 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:36.081860 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.081822 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7"] Jul 05 19:38:36.082359 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.082103 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" podUID="b8a7b821-d433-4572-868b-1d736ad77056" containerName="manager" containerID="cri-o://691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade" gracePeriod=2 Jul 05 19:38:36.084689 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.084666 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:38:36.106865 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.106835 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7"] Jul 05 19:38:36.107007 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.106869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.107007 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.106892 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:38:36.107646 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.107621 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8"] Jul 05 19:38:36.108436 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.108381 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" podUID="1689be87-04f2-421c-8707-020ad45b02d5" containerName="manager" containerID="cri-o://cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd" gracePeriod=2 Jul 05 19:38:36.110910 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:36.110759 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:36.120965 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.120942 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8"] Jul 05 19:38:36.135880 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.135858 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh"] Jul 05 19:38:36.136393 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136369 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1689be87-04f2-421c-8707-020ad45b02d5" containerName="manager" Jul 05 19:38:36.136393 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136393 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1689be87-04f2-421c-8707-020ad45b02d5" containerName="manager" Jul 05 19:38:36.136563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136414 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a7b821-d433-4572-868b-1d736ad77056" containerName="manager" Jul 05 19:38:36.136563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136423 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a7b821-d433-4572-868b-1d736ad77056" containerName="manager" Jul 05 19:38:36.136563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136510 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a7b821-d433-4572-868b-1d736ad77056" containerName="manager" Jul 05 19:38:36.136563 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.136531 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1689be87-04f2-421c-8707-020ad45b02d5" containerName="manager" Jul 05 19:38:36.141397 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.141380 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:36.146904 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:36.146007 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:36.148398 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:36.148368 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:36.149686 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.149665 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh"] Jul 05 19:38:36.162018 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.161991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.162122 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.162096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxp4\" (UniqueName: \"kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.263358 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.263328 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.263492 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.263421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxp4\" (UniqueName: \"kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.263492 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.263461 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xkrb\" (UniqueName: \"kubernetes.io/projected/b63306a6-fd48-4fa2-9ea8-c2970c818d7d-kube-api-access-8xkrb\") pod \"limitador-operator-controller-manager-859d7b78bd-dclxh\" (UID: \"b63306a6-fd48-4fa2-9ea8-c2970c818d7d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:36.263744 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.263724 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.272511 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.272485 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxp4\" (UniqueName: \"kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-45n9t\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.364516 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.364445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xkrb\" (UniqueName: \"kubernetes.io/projected/b63306a6-fd48-4fa2-9ea8-c2970c818d7d-kube-api-access-8xkrb\") pod \"limitador-operator-controller-manager-859d7b78bd-dclxh\" (UID: \"b63306a6-fd48-4fa2-9ea8-c2970c818d7d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:36.380808 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.380761 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xkrb\" (UniqueName: \"kubernetes.io/projected/b63306a6-fd48-4fa2-9ea8-c2970c818d7d-kube-api-access-8xkrb\") pod \"limitador-operator-controller-manager-859d7b78bd-dclxh\" (UID: \"b63306a6-fd48-4fa2-9ea8-c2970c818d7d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:36.447687 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.447650 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6948588c6-77hhr" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" containerID="cri-o://7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c" gracePeriod=15 Jul 05 19:38:36.503734 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.503709 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:36.510921 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:36.510901 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:37.235227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:37.235187 2581 patch_prober.go:28] interesting pod/console-6948588c6-77hhr container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused" start-of-body= Jul 05 19:38:37.235612 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:37.235246 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6948588c6-77hhr" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" probeResult="failure" output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused" Jul 05 19:38:47.234518 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:47.234482 2581 patch_prober.go:28] interesting pod/console-6948588c6-77hhr container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused" start-of-body= Jul 05 19:38:47.234909 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:47.234549 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6948588c6-77hhr" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" probeResult="failure" output="Get \"https://10.133.0.21:8443/health\": dial tcp 10.133.0.21:8443: connect: connection refused" Jul 05 19:38:48.501229 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:48.501027 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74c9bb5_6b07_4770_809d_da440c09daa3.slice/crio-2fac7179d2803c9dd35abe296142c5e4eb7481c2060120da817b0848073f8be0 WatchSource:0}: Error finding container 2fac7179d2803c9dd35abe296142c5e4eb7481c2060120da817b0848073f8be0: Status 404 returned error can't find the container with id 2fac7179d2803c9dd35abe296142c5e4eb7481c2060120da817b0848073f8be0 Jul 05 19:38:48.503697 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.501698 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:38:48.504456 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.504213 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:38:48.506748 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.506730 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.512651 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.512631 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.525786 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.525747 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.527973 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.527945 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.532254 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.532229 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh"] Jul 05 19:38:48.534394 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:38:48.534369 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63306a6_fd48_4fa2_9ea8_c2970c818d7d.slice/crio-d9a27ae063ffa28840d65d84d7d6ad19db306ad4bcb704a8eeb0abcaa814cab2 WatchSource:0}: Error finding container d9a27ae063ffa28840d65d84d7d6ad19db306ad4bcb704a8eeb0abcaa814cab2: Status 404 returned error can't find the container with id d9a27ae063ffa28840d65d84d7d6ad19db306ad4bcb704a8eeb0abcaa814cab2 Jul 05 19:38:48.585477 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.585381 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" event={"ID":"b74c9bb5-6b07-4770-809d-da440c09daa3","Type":"ContainerStarted","Data":"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84"} Jul 05 19:38:48.585477 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.585434 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" event={"ID":"b74c9bb5-6b07-4770-809d-da440c09daa3","Type":"ContainerStarted","Data":"2fac7179d2803c9dd35abe296142c5e4eb7481c2060120da817b0848073f8be0"} Jul 05 19:38:48.585630 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.585497 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:48.586935 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.586914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" event={"ID":"b63306a6-fd48-4fa2-9ea8-c2970c818d7d","Type":"ContainerStarted","Data":"d9a27ae063ffa28840d65d84d7d6ad19db306ad4bcb704a8eeb0abcaa814cab2"} Jul 05 19:38:48.587164 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.586988 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:38:48.587634 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.587610 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.588299 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588278 2581 generic.go:356] "Generic (PLEG): container finished" podID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerID="7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c" exitCode=0 Jul 05 19:38:48.588390 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588321 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6948588c6-77hhr" Jul 05 19:38:48.588390 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588351 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6948588c6-77hhr" event={"ID":"63f73c4d-b85f-4aba-86fa-ca0e8996acf4","Type":"ContainerDied","Data":"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c"} Jul 05 19:38:48.588493 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6948588c6-77hhr" event={"ID":"63f73c4d-b85f-4aba-86fa-ca0e8996acf4","Type":"ContainerDied","Data":"d7bf5a1d4132f57d31b9e2668ac1a4da186cd1be55753b67876f271585617169"} Jul 05 19:38:48.588493 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588438 2581 scope.go:117] "RemoveContainer" containerID="7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c" Jul 05 19:38:48.588962 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.588938 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589049 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589015 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589049 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589043 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589187 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589101 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589187 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589136 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87kgv\" (UniqueName: \"kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv\") pod \"1689be87-04f2-421c-8707-020ad45b02d5\" (UID: \"1689be87-04f2-421c-8707-020ad45b02d5\") " Jul 05 19:38:48.589187 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589179 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589228 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl7rk\" (UniqueName: \"kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk\") pod \"b8a7b821-d433-4572-868b-1d736ad77056\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " Jul 05 19:38:48.589339 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589290 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjbzt\" (UniqueName: \"kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt\") pod \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\" (UID: \"63f73c4d-b85f-4aba-86fa-ca0e8996acf4\") " Jul 05 19:38:48.589487 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589324 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume\") pod \"b8a7b821-d433-4572-868b-1d736ad77056\" (UID: \"b8a7b821-d433-4572-868b-1d736ad77056\") " Jul 05 19:38:48.589487 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589385 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config" (OuterVolumeSpecName: "console-config") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:38:48.589487 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589456 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca" (OuterVolumeSpecName: "service-ca") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:38:48.589704 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589590 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.589765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589607 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-service-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.589885 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589861 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:38:48.590003 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.589965 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8a7b821-d433-4572-868b-1d736ad77056" (UID: "b8a7b821-d433-4572-868b-1d736ad77056"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:38:48.590148 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.590121 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.590421 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.590356 2581 generic.go:356] "Generic (PLEG): container finished" podID="1689be87-04f2-421c-8707-020ad45b02d5" containerID="cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd" exitCode=0 Jul 05 19:38:48.590500 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.590485 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.590571 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.590524 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:38:48.592678 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.592652 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" event={"ID":"efbfc0c7-6728-4e44-8e8e-4331657ccf06","Type":"ContainerStarted","Data":"76cc5cea9787f535ff2f5b03e96e38ef1451c901f7f69d2ad1a8730db24d986e"} Jul 05 19:38:48.593802 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.593777 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:38:48.593984 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.593848 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:38:48.593984 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.593922 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv" (OuterVolumeSpecName: "kube-api-access-87kgv") pod "1689be87-04f2-421c-8707-020ad45b02d5" (UID: "1689be87-04f2-421c-8707-020ad45b02d5"). InnerVolumeSpecName "kube-api-access-87kgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:38:48.594209 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.593993 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt" (OuterVolumeSpecName: "kube-api-access-qjbzt") pod "63f73c4d-b85f-4aba-86fa-ca0e8996acf4" (UID: "63f73c4d-b85f-4aba-86fa-ca0e8996acf4"). InnerVolumeSpecName "kube-api-access-qjbzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:38:48.594400 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.594361 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk" (OuterVolumeSpecName: "kube-api-access-gl7rk") pod "b8a7b821-d433-4572-868b-1d736ad77056" (UID: "b8a7b821-d433-4572-868b-1d736ad77056"). InnerVolumeSpecName "kube-api-access-gl7rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:38:48.594791 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.594772 2581 generic.go:356] "Generic (PLEG): container finished" podID="b8a7b821-d433-4572-868b-1d736ad77056" containerID="691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade" exitCode=0 Jul 05 19:38:48.594866 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.594816 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.601861 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.601842 2581 scope.go:117] "RemoveContainer" containerID="7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c" Jul 05 19:38:48.602309 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.602287 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c\": container with ID starting with 7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c not found: ID does not exist" containerID="7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c" Jul 05 19:38:48.602396 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.602318 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c"} err="failed to get container status \"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c\": rpc error: code = NotFound desc = could not find container \"7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c\": container with ID starting with 7aa6fa5767ba4bf5efb40267436222bc4176f99b1e4a8196ab70a6f6f4fb610c not found: ID does not exist" Jul 05 19:38:48.602396 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.602341 2581 scope.go:117] "RemoveContainer" containerID="cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd" Jul 05 19:38:48.609849 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.609790 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" podStartSLOduration=12.609776618 podStartE2EDuration="12.609776618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:48.608029203 +0000 UTC m=+529.063812633" watchObservedRunningTime="2026-07-05 19:38:48.609776618 +0000 UTC m=+529.065560048" Jul 05 19:38:48.610636 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.610609 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.611365 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.611348 2581 scope.go:117] "RemoveContainer" containerID="cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd" Jul 05 19:38:48.611711 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.611690 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd\": container with ID starting with cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd not found: ID does not exist" containerID="cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd" Jul 05 19:38:48.611765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.611719 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd"} err="failed to get container status \"cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd\": rpc error: code = NotFound desc = could not find container \"cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd\": container with ID starting with cca9dd75e10d8b943b481184a9332259c48d57ba08c1ff4090c0451245ce8bfd not found: ID does not exist" Jul 05 19:38:48.611765 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.611735 2581 scope.go:117] "RemoveContainer" containerID="691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade" Jul 05 19:38:48.619888 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.619872 2581 scope.go:117] "RemoveContainer" containerID="691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade" Jul 05 19:38:48.620151 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.620134 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade\": container with ID starting with 691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade not found: ID does not exist" containerID="691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade" Jul 05 19:38:48.620204 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.620158 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade"} err="failed to get container status \"691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade\": rpc error: code = NotFound desc = could not find container \"691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade\": container with ID starting with 691ce6f5066bb9810880f321dea360aa9bb5675dd78b4d1d2e57e1d76e38fade not found: ID does not exist" Jul 05 19:38:48.630696 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.630657 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bbnj8" podStartSLOduration=2.014249837 podStartE2EDuration="23.63064569s" totalImagesPullingTime="21.616395853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:48.629779635 +0000 UTC m=+529.085563066" watchObservedRunningTime="2026-07-05 19:38:48.63064569 +0000 UTC m=+529.086429131" Jul 05 19:38:48.631890 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.631864 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.650742 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.650704 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" podStartSLOduration=12.650691984 podStartE2EDuration="12.650691984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:48.649120979 +0000 UTC m=+529.104904411" watchObservedRunningTime="2026-07-05 19:38:48.650691984 +0000 UTC m=+529.106475415" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691145 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-oauth-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691173 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kgv\" (UniqueName: \"kubernetes.io/projected/1689be87-04f2-421c-8707-020ad45b02d5-kube-api-access-87kgv\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691188 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691202 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-serving-cert\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691216 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl7rk\" (UniqueName: \"kubernetes.io/projected/b8a7b821-d433-4572-868b-1d736ad77056-kube-api-access-gl7rk\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691229 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjbzt\" (UniqueName: \"kubernetes.io/projected/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-kube-api-access-qjbzt\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691245 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a7b821-d433-4572-868b-1d736ad77056-extensions-socket-volume\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.691282 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.691259 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63f73c4d-b85f-4aba-86fa-ca0e8996acf4-console-oauth-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:38:48.899741 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.899709 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.901930 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.901903 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:48.915453 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.915428 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:38:48.919263 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.919232 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:48.919736 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:48.919705 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6948588c6-77hhr"] Jul 05 19:38:48.922398 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:48.922376 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:49.602588 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:49.602491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" event={"ID":"b63306a6-fd48-4fa2-9ea8-c2970c818d7d","Type":"ContainerStarted","Data":"1e1e5e44304e65bc221f60254209723b4db1fb7a0d12618fd1cf48a45dfdfa99"} Jul 05 19:38:50.061300 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:50.061266 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-g6lf7\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="b8a7b821-d433-4572-868b-1d736ad77056" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-g6lf7" Jul 05 19:38:50.061538 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:50.061519 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1689be87-04f2-421c-8707-020ad45b02d5" path="/var/lib/kubelet/pods/1689be87-04f2-421c-8707-020ad45b02d5/volumes" Jul 05 19:38:50.061893 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:50.061881 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" path="/var/lib/kubelet/pods/63f73c4d-b85f-4aba-86fa-ca0e8996acf4/volumes" Jul 05 19:38:50.062204 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:50.062192 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a7b821-d433-4572-868b-1d736ad77056" path="/var/lib/kubelet/pods/b8a7b821-d433-4572-868b-1d736ad77056/volumes" Jul 05 19:38:50.064269 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:38:50.064248 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dsxq8\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" podUID="1689be87-04f2-421c-8707-020ad45b02d5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dsxq8" Jul 05 19:38:59.607571 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.607494 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:38:59.902445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.902417 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:38:59.902824 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.902810 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" Jul 05 19:38:59.902866 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.902827 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" Jul 05 19:38:59.902912 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.902901 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f73c4d-b85f-4aba-86fa-ca0e8996acf4" containerName="console" Jul 05 19:38:59.905516 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.905502 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:38:59.922412 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:38:59.922380 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:39:00.089517 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.089486 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.089705 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.089586 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgkm\" (UniqueName: \"kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.190945 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.190904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgkm\" (UniqueName: \"kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.191156 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.190985 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.191412 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.191391 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.200803 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.200778 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgkm\" (UniqueName: \"kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm\") pod \"kuadrant-operator-controller-manager-86878c769f-tl2m8\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.215624 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.215599 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.356710 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.356679 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:39:00.359957 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:39:00.359933 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bea77f6_4169_4adb_9c16_742fcd3f190b.slice/crio-d5f4eb0a8b81d52d6e93d739fc2dabca42927d73eb4a41282563e528877fb049 WatchSource:0}: Error finding container d5f4eb0a8b81d52d6e93d739fc2dabca42927d73eb4a41282563e528877fb049: Status 404 returned error can't find the container with id d5f4eb0a8b81d52d6e93d739fc2dabca42927d73eb4a41282563e528877fb049 Jul 05 19:39:00.613880 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.613793 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dclxh" Jul 05 19:39:00.668025 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.667990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" event={"ID":"7bea77f6-4169-4adb-9c16-742fcd3f190b","Type":"ContainerStarted","Data":"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c"} Jul 05 19:39:00.668231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.668048 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" event={"ID":"7bea77f6-4169-4adb-9c16-742fcd3f190b","Type":"ContainerStarted","Data":"d5f4eb0a8b81d52d6e93d739fc2dabca42927d73eb4a41282563e528877fb049"} Jul 05 19:39:00.668231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.668097 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:00.705466 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:00.705414 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" podStartSLOduration=1.70539749 podStartE2EDuration="1.70539749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:00.702081699 +0000 UTC m=+541.157865124" watchObservedRunningTime="2026-07-05 19:39:00.70539749 +0000 UTC m=+541.161180920" Jul 05 19:39:11.676659 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:11.676623 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:39:11.729785 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:11.729754 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:39:11.730001 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:11.729960 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" podUID="b74c9bb5-6b07-4770-809d-da440c09daa3" containerName="manager" containerID="cri-o://8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84" gracePeriod=10 Jul 05 19:39:12.571559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.571533 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:39:12.599028 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.599003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume\") pod \"b74c9bb5-6b07-4770-809d-da440c09daa3\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " Jul 05 19:39:12.599197 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.599112 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crxp4\" (UniqueName: \"kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4\") pod \"b74c9bb5-6b07-4770-809d-da440c09daa3\" (UID: \"b74c9bb5-6b07-4770-809d-da440c09daa3\") " Jul 05 19:39:12.599436 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.599413 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b74c9bb5-6b07-4770-809d-da440c09daa3" (UID: "b74c9bb5-6b07-4770-809d-da440c09daa3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:39:12.601363 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.601341 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4" (OuterVolumeSpecName: "kube-api-access-crxp4") pod "b74c9bb5-6b07-4770-809d-da440c09daa3" (UID: "b74c9bb5-6b07-4770-809d-da440c09daa3"). InnerVolumeSpecName "kube-api-access-crxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:12.700241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.700159 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b74c9bb5-6b07-4770-809d-da440c09daa3-extensions-socket-volume\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:39:12.700241 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.700193 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxp4\" (UniqueName: \"kubernetes.io/projected/b74c9bb5-6b07-4770-809d-da440c09daa3-kube-api-access-crxp4\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:39:12.731879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.731846 2581 generic.go:356] "Generic (PLEG): container finished" podID="b74c9bb5-6b07-4770-809d-da440c09daa3" containerID="8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84" exitCode=0 Jul 05 19:39:12.732035 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.731891 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" Jul 05 19:39:12.732035 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.731931 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" event={"ID":"b74c9bb5-6b07-4770-809d-da440c09daa3","Type":"ContainerDied","Data":"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84"} Jul 05 19:39:12.732035 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.731988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t" event={"ID":"b74c9bb5-6b07-4770-809d-da440c09daa3","Type":"ContainerDied","Data":"2fac7179d2803c9dd35abe296142c5e4eb7481c2060120da817b0848073f8be0"} Jul 05 19:39:12.732035 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.732017 2581 scope.go:117] "RemoveContainer" containerID="8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84" Jul 05 19:39:12.741661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.741637 2581 scope.go:117] "RemoveContainer" containerID="8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84" Jul 05 19:39:12.741956 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:39:12.741923 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84\": container with ID starting with 8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84 not found: ID does not exist" containerID="8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84" Jul 05 19:39:12.742026 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.741965 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84"} err="failed to get container status \"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84\": rpc error: code = NotFound desc = could not find container \"8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84\": container with ID starting with 8777eeaa342d9c405e17a553c984ec5e4486bd8b0440ed5a38a13a8f1aabcd84 not found: ID does not exist" Jul 05 19:39:12.754929 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.754896 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:39:12.760834 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:12.760799 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-45n9t"] Jul 05 19:39:14.060755 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:14.060724 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74c9bb5-6b07-4770-809d-da440c09daa3" path="/var/lib/kubelet/pods/b74c9bb5-6b07-4770-809d-da440c09daa3/volumes" Jul 05 19:39:20.116660 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.116627 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:39:20.117036 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.117003 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74c9bb5-6b07-4770-809d-da440c09daa3" containerName="manager" Jul 05 19:39:20.117036 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.117014 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74c9bb5-6b07-4770-809d-da440c09daa3" containerName="manager" Jul 05 19:39:20.117147 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.117106 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74c9bb5-6b07-4770-809d-da440c09daa3" containerName="manager" Jul 05 19:39:20.207346 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.207310 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:39:20.207346 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.207347 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.207570 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.207370 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:39:20.210010 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.209991 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 05 19:39:20.266273 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.266250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7dv\" (UniqueName: \"kubernetes.io/projected/dea16eac-389d-4e50-9bf5-f3bedb634720-kube-api-access-pf7dv\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.266372 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.266343 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dea16eac-389d-4e50-9bf5-f3bedb634720-config-file\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.366914 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.366835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7dv\" (UniqueName: \"kubernetes.io/projected/dea16eac-389d-4e50-9bf5-f3bedb634720-kube-api-access-pf7dv\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.367034 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.366928 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dea16eac-389d-4e50-9bf5-f3bedb634720-config-file\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.367590 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.367570 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dea16eac-389d-4e50-9bf5-f3bedb634720-config-file\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.375646 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.375613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7dv\" (UniqueName: \"kubernetes.io/projected/dea16eac-389d-4e50-9bf5-f3bedb634720-kube-api-access-pf7dv\") pod \"limitador-limitador-54d6d988d5-85wpn\" (UID: \"dea16eac-389d-4e50-9bf5-f3bedb634720\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.517904 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.517870 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:20.658129 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.658092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:39:20.661188 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:39:20.661164 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea16eac_389d_4e50_9bf5_f3bedb634720.slice/crio-f22353d20cfd0c7c95f76ca975a868895694105b95f816e76c9c92dfa1fc2509 WatchSource:0}: Error finding container f22353d20cfd0c7c95f76ca975a868895694105b95f816e76c9c92dfa1fc2509: Status 404 returned error can't find the container with id f22353d20cfd0c7c95f76ca975a868895694105b95f816e76c9c92dfa1fc2509 Jul 05 19:39:20.785399 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:20.785370 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" event={"ID":"dea16eac-389d-4e50-9bf5-f3bedb634720","Type":"ContainerStarted","Data":"f22353d20cfd0c7c95f76ca975a868895694105b95f816e76c9c92dfa1fc2509"} Jul 05 19:39:23.806073 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:23.806022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" event={"ID":"dea16eac-389d-4e50-9bf5-f3bedb634720","Type":"ContainerStarted","Data":"f77a372d99732d224177c6020ef0bd716deaa81d8415517f4e78f0f36aee826f"} Jul 05 19:39:23.806507 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:23.806120 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:39:23.823149 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:23.823108 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" podStartSLOduration=1.366210046 podStartE2EDuration="3.823095873s" totalImagesPullingTime="2.456885827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:23.821373467 +0000 UTC m=+564.277156896" watchObservedRunningTime="2026-07-05 19:39:23.823095873 +0000 UTC m=+564.278879303" Jul 05 19:39:34.812866 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:39:34.812836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-85wpn" Jul 05 19:40:00.002362 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:00.002323 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:40:00.006423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:00.006399 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:40:36.987181 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:36.987148 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t26cp"] Jul 05 19:40:36.990848 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:36.990828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:36.993910 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:36.993887 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 05 19:40:36.994044 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:36.993957 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-njz2x\"" Jul 05 19:40:37.000445 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.000425 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t26cp"] Jul 05 19:40:37.147954 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.147914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grrj\" (UniqueName: \"kubernetes.io/projected/8ca415f0-3600-4e50-8c56-72135202d508-kube-api-access-6grrj\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.148166 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.148033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca415f0-3600-4e50-8c56-72135202d508-plugins-config-volume\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.238336 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.238250 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:40:37.249094 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.249035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grrj\" (UniqueName: \"kubernetes.io/projected/8ca415f0-3600-4e50-8c56-72135202d508-kube-api-access-6grrj\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.249269 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.249160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca415f0-3600-4e50-8c56-72135202d508-plugins-config-volume\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.249892 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.249873 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca415f0-3600-4e50-8c56-72135202d508-plugins-config-volume\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.256913 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.256885 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grrj\" (UniqueName: \"kubernetes.io/projected/8ca415f0-3600-4e50-8c56-72135202d508-kube-api-access-6grrj\") pod \"payload-processing-58f86c7798-t26cp\" (UID: \"8ca415f0-3600-4e50-8c56-72135202d508\") " pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.302384 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.302350 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:37.434886 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.434858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t26cp"] Jul 05 19:40:37.436531 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:40:37.436504 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca415f0_3600_4e50_8c56_72135202d508.slice/crio-9fd7013d88ea56e714b7c334b03dd6b322702795606776e629ac9a93a312aad0 WatchSource:0}: Error finding container 9fd7013d88ea56e714b7c334b03dd6b322702795606776e629ac9a93a312aad0: Status 404 returned error can't find the container with id 9fd7013d88ea56e714b7c334b03dd6b322702795606776e629ac9a93a312aad0 Jul 05 19:40:37.438530 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:37.438511 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:40:38.211821 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:38.211787 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" event={"ID":"8ca415f0-3600-4e50-8c56-72135202d508","Type":"ContainerStarted","Data":"9fd7013d88ea56e714b7c334b03dd6b322702795606776e629ac9a93a312aad0"} Jul 05 19:40:39.220227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:39.220197 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" event={"ID":"8ca415f0-3600-4e50-8c56-72135202d508","Type":"ContainerStarted","Data":"ecfbb8e3191b73d8015393d67b2ad85482e5319209342ae39db4f8dcc62c8969"} Jul 05 19:40:39.220609 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:39.220317 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:39.236614 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:39.236541 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" podStartSLOduration=1.990575851 podStartE2EDuration="3.236521207s" totalImagesPullingTime="1.245945356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:39.235229059 +0000 UTC m=+639.691012488" watchObservedRunningTime="2026-07-05 19:40:39.236521207 +0000 UTC m=+639.692304618" Jul 05 19:40:50.227546 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:50.227517 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-t26cp" Jul 05 19:40:51.735389 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:51.735353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:40:59.735352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:40:59.735315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:41:14.735661 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:14.735632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:41:18.644559 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.644530 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb"] Jul 05 19:41:18.648614 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.648598 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.652829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.652541 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 05 19:41:18.652829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.652560 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 05 19:41:18.652829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.652575 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cxv76\"" Jul 05 19:41:18.652829 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.652590 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 05 19:41:18.657196 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.657177 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb"] Jul 05 19:41:18.816728 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816691 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7439118-a02c-4c16-b210-39ba14a46266-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.816902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.816902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816792 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.816902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816822 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f7j9\" (UniqueName: \"kubernetes.io/projected/d7439118-a02c-4c16-b210-39ba14a46266-kube-api-access-7f7j9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.816902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816843 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.816902 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.816872 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.917747 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.917712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f7j9\" (UniqueName: \"kubernetes.io/projected/d7439118-a02c-4c16-b210-39ba14a46266-kube-api-access-7f7j9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.917928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.917756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.917928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.917782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.917928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.917834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7439118-a02c-4c16-b210-39ba14a46266-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.917928 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.917863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.918180 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.918047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.918180 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.918151 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.918288 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.918211 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.918288 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.918238 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.920280 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.920252 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7439118-a02c-4c16-b210-39ba14a46266-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.920565 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.920544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7439118-a02c-4c16-b210-39ba14a46266-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.925709 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.925689 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f7j9\" (UniqueName: \"kubernetes.io/projected/d7439118-a02c-4c16-b210-39ba14a46266-kube-api-access-7f7j9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb\" (UID: \"d7439118-a02c-4c16-b210-39ba14a46266\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:18.959881 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:18.959863 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:19.095423 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:19.095397 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb"] Jul 05 19:41:19.097306 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:41:19.097278 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7439118_a02c_4c16_b210_39ba14a46266.slice/crio-d236d0281e08fbdc8e2fdd82fa6459e7533b4852528bb2df23f0e9452f7d7736 WatchSource:0}: Error finding container d236d0281e08fbdc8e2fdd82fa6459e7533b4852528bb2df23f0e9452f7d7736: Status 404 returned error can't find the container with id d236d0281e08fbdc8e2fdd82fa6459e7533b4852528bb2df23f0e9452f7d7736 Jul 05 19:41:19.435048 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:19.434958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" event={"ID":"d7439118-a02c-4c16-b210-39ba14a46266","Type":"ContainerStarted","Data":"d236d0281e08fbdc8e2fdd82fa6459e7533b4852528bb2df23f0e9452f7d7736"} Jul 05 19:41:20.441793 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:20.441756 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" event={"ID":"d7439118-a02c-4c16-b210-39ba14a46266","Type":"ContainerStarted","Data":"6c03a8e8f4b98e80f07b6fa5e9bbbae237a7f90b18e9c49ea47b3405514e740d"} Jul 05 19:41:20.460267 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:20.460219 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" podStartSLOduration=1.434407437 podStartE2EDuration="2.460204221s" totalImagesPullingTime="1.025796784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:41:20.456825914 +0000 UTC m=+680.912609345" watchObservedRunningTime="2026-07-05 19:41:20.460204221 +0000 UTC m=+680.915987656" Jul 05 19:41:20.641538 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:20.641510 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:41:28.960691 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:28.960655 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:28.961081 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:28.960741 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:28.973149 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:28.973124 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:29.501022 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:29.500995 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb" Jul 05 19:41:31.142609 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:31.142579 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:41:54.048109 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.047991 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b"] Jul 05 19:41:54.051457 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.051440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.054224 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.054204 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 05 19:41:54.065044 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.065018 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b"] Jul 05 19:41:54.121181 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121141 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.121352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkkx\" (UniqueName: \"kubernetes.io/projected/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-kube-api-access-qhkkx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.121352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.121352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.121352 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121336 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.121491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.121417 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222188 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222216 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkkx\" (UniqueName: \"kubernetes.io/projected/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-kube-api-access-qhkkx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222391 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222659 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222641 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222717 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222696 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.222793 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.222777 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.224744 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.224722 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.225022 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.225003 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.229578 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.229557 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkkx\" (UniqueName: \"kubernetes.io/projected/d0879cc8-a2cb-4212-8dbe-b790cc0f1998-kube-api-access-qhkkx\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b\" (UID: \"d0879cc8-a2cb-4212-8dbe-b790cc0f1998\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.363787 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.363680 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:41:54.509554 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.509525 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b"] Jul 05 19:41:54.511881 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:41:54.511853 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0879cc8_a2cb_4212_8dbe_b790cc0f1998.slice/crio-545c24c2f394444fbba7193ae28719d81b0a864ced8d0837d3dcfbe75d9922f7 WatchSource:0}: Error finding container 545c24c2f394444fbba7193ae28719d81b0a864ced8d0837d3dcfbe75d9922f7: Status 404 returned error can't find the container with id 545c24c2f394444fbba7193ae28719d81b0a864ced8d0837d3dcfbe75d9922f7 Jul 05 19:41:54.632841 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:54.632749 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" event={"ID":"d0879cc8-a2cb-4212-8dbe-b790cc0f1998","Type":"ContainerStarted","Data":"545c24c2f394444fbba7193ae28719d81b0a864ced8d0837d3dcfbe75d9922f7"} Jul 05 19:41:55.639943 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:55.639904 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" event={"ID":"d0879cc8-a2cb-4212-8dbe-b790cc0f1998","Type":"ContainerStarted","Data":"0527a21d7689efe0514e59528cde7d5cbf8e8220ec1ac6aa363f2691e9f62297"} Jul 05 19:41:55.655522 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:55.655469 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" podStartSLOduration=1.494023038 podStartE2EDuration="1.655449656s" totalImagesPullingTime="161.426618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:41:55.654831305 +0000 UTC m=+716.110614736" watchObservedRunningTime="2026-07-05 19:41:55.655449656 +0000 UTC m=+716.111233087" Jul 05 19:41:56.444549 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:41:56.444508 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:42:04.364130 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:42:04.364082 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:42:04.364614 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:42:04.364145 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:42:04.376842 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:42:04.376819 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:42:04.699263 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:42:04.699231 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b" Jul 05 19:43:28.344615 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:43:28.344535 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:43:40.738882 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:43:40.738850 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:43:49.040155 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:43:49.040120 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:43:57.636517 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:43:57.635695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:44:07.639508 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:44:07.639473 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:44:37.451584 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:44:37.451553 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:44:48.747572 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:44:48.747535 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:45:00.040475 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:00.040404 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:45:00.041834 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:00.041815 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:45:22.698209 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:22.698159 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:45:32.735344 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:32.735311 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:45:40.941715 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:40.941686 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:45:50.237505 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:50.237472 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:45:59.937377 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:45:59.937343 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:46:10.244382 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:46:10.244348 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:46:21.635458 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:46:21.635378 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:46:31.435169 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:46:31.435136 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:46:53.540389 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:46:53.540348 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:47:04.548011 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:47:04.547971 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:47:27.435231 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:47:27.435191 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:47:37.040997 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:47:37.040961 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:47:54.846631 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:47:54.846557 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:48:03.840451 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:03.840411 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:48:13.043925 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:13.043884 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:48:22.240845 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:22.240804 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:48:30.543828 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:30.543786 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:48:57.995125 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:57.995089 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:48:57.995633 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:57.995276 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" podUID="7bea77f6-4169-4adb-9c16-742fcd3f190b" containerName="manager" containerID="cri-o://7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c" gracePeriod=10 Jul 05 19:48:59.757392 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.757370 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:48:59.925419 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.925383 2581 generic.go:356] "Generic (PLEG): container finished" podID="7bea77f6-4169-4adb-9c16-742fcd3f190b" containerID="7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c" exitCode=0 Jul 05 19:48:59.925604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.925437 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" Jul 05 19:48:59.925604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.925472 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" event={"ID":"7bea77f6-4169-4adb-9c16-742fcd3f190b","Type":"ContainerDied","Data":"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c"} Jul 05 19:48:59.925604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.925526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8" event={"ID":"7bea77f6-4169-4adb-9c16-742fcd3f190b","Type":"ContainerDied","Data":"d5f4eb0a8b81d52d6e93d739fc2dabca42927d73eb4a41282563e528877fb049"} Jul 05 19:48:59.925604 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.925550 2581 scope.go:117] "RemoveContainer" containerID="7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c" Jul 05 19:48:59.932726 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.932705 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume\") pod \"7bea77f6-4169-4adb-9c16-742fcd3f190b\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " Jul 05 19:48:59.932879 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.932740 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wgkm\" (UniqueName: \"kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm\") pod \"7bea77f6-4169-4adb-9c16-742fcd3f190b\" (UID: \"7bea77f6-4169-4adb-9c16-742fcd3f190b\") " Jul 05 19:48:59.933151 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.933128 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7bea77f6-4169-4adb-9c16-742fcd3f190b" (UID: "7bea77f6-4169-4adb-9c16-742fcd3f190b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:48:59.934980 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.934902 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm" (OuterVolumeSpecName: "kube-api-access-2wgkm") pod "7bea77f6-4169-4adb-9c16-742fcd3f190b" (UID: "7bea77f6-4169-4adb-9c16-742fcd3f190b"). InnerVolumeSpecName "kube-api-access-2wgkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:48:59.935124 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.934980 2581 scope.go:117] "RemoveContainer" containerID="7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c" Jul 05 19:48:59.935356 ip-10-0-143-177 kubenswrapper[2581]: E0705 19:48:59.935339 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c\": container with ID starting with 7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c not found: ID does not exist" containerID="7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c" Jul 05 19:48:59.935422 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:48:59.935367 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c"} err="failed to get container status \"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c\": rpc error: code = NotFound desc = could not find container \"7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c\": container with ID starting with 7a288c5e17eadc3de0102ac78fbc208a1e98e94fbf9a0642c2d4e6901177580c not found: ID does not exist" Jul 05 19:49:00.033449 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:00.033416 2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7bea77f6-4169-4adb-9c16-742fcd3f190b-extensions-socket-volume\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:49:00.033449 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:00.033445 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgkm\" (UniqueName: \"kubernetes.io/projected/7bea77f6-4169-4adb-9c16-742fcd3f190b-kube-api-access-2wgkm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 19:49:00.241522 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:00.241448 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:49:00.245107 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:00.245082 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tl2m8"] Jul 05 19:49:02.062291 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:02.062260 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bea77f6-4169-4adb-9c16-742fcd3f190b" path="/var/lib/kubelet/pods/7bea77f6-4169-4adb-9c16-742fcd3f190b/volumes" Jul 05 19:49:59.120291 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.120257 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44"] Jul 05 19:49:59.120771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.120619 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bea77f6-4169-4adb-9c16-742fcd3f190b" containerName="manager" Jul 05 19:49:59.120771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.120630 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bea77f6-4169-4adb-9c16-742fcd3f190b" containerName="manager" Jul 05 19:49:59.120771 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.120704 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bea77f6-4169-4adb-9c16-742fcd3f190b" containerName="manager" Jul 05 19:49:59.123743 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.123724 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.126561 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.126543 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wk9wk\"" Jul 05 19:49:59.137097 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.137076 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44"] Jul 05 19:49:59.203978 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.203947 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee2f7887-7aca-4045-a1b1-e47b181ec56a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.204138 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.204025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphzl\" (UniqueName: \"kubernetes.io/projected/ee2f7887-7aca-4045-a1b1-e47b181ec56a-kube-api-access-wphzl\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.305460 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.305430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphzl\" (UniqueName: \"kubernetes.io/projected/ee2f7887-7aca-4045-a1b1-e47b181ec56a-kube-api-access-wphzl\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.305590 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.305511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee2f7887-7aca-4045-a1b1-e47b181ec56a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.305833 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.305815 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee2f7887-7aca-4045-a1b1-e47b181ec56a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.317574 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.317552 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphzl\" (UniqueName: \"kubernetes.io/projected/ee2f7887-7aca-4045-a1b1-e47b181ec56a-kube-api-access-wphzl\") pod \"kuadrant-operator-controller-manager-86878c769f-2sv44\" (UID: \"ee2f7887-7aca-4045-a1b1-e47b181ec56a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.434028 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.434003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:49:59.572796 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:49:59.572770 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44"] Jul 05 19:49:59.575788 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:49:59.575760 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2f7887_7aca_4045_a1b1_e47b181ec56a.slice/crio-c08e4a1b4e6942f91050106a521fedb212330cee988709cbe9ffa5ec1762d473 WatchSource:0}: Error finding container c08e4a1b4e6942f91050106a521fedb212330cee988709cbe9ffa5ec1762d473: Status 404 returned error can't find the container with id c08e4a1b4e6942f91050106a521fedb212330cee988709cbe9ffa5ec1762d473 Jul 05 19:50:00.078647 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.078617 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:50:00.079186 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.079169 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:50:00.237454 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.237421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" event={"ID":"ee2f7887-7aca-4045-a1b1-e47b181ec56a","Type":"ContainerStarted","Data":"bec6041ba0a6f596b9b745fb967d37df6c7ed037a53074c4dbdff9eab1645942"} Jul 05 19:50:00.237870 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.237481 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:50:00.237870 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.237500 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" event={"ID":"ee2f7887-7aca-4045-a1b1-e47b181ec56a","Type":"ContainerStarted","Data":"c08e4a1b4e6942f91050106a521fedb212330cee988709cbe9ffa5ec1762d473"} Jul 05 19:50:00.257814 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:00.257759 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" podStartSLOduration=1.257740978 podStartE2EDuration="1.257740978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:50:00.25567558 +0000 UTC m=+1200.711459012" watchObservedRunningTime="2026-07-05 19:50:00.257740978 +0000 UTC m=+1200.713524405" Jul 05 19:50:11.246851 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:11.246812 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2sv44" Jul 05 19:50:20.945543 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:20.945499 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:50:31.546369 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:31.546331 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:50:42.342491 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:42.342458 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:50:53.247718 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:50:53.247638 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:03.835213 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:03.835178 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:14.835516 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:14.835474 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:23.640497 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:23.640465 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:36.437489 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:36.437451 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:45.630644 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:45.630607 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:51:55.243490 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:51:55.243454 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:52:08.346227 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:52:08.346194 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:52:26.843478 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:52:26.843397 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:52:35.738582 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:52:35.738546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:52:46.468540 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:52:46.468497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:52:57.237834 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:52:57.237799 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:11.639838 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:11.639805 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:21.140614 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:21.140581 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:30.043752 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:30.043722 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:40.236468 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:40.236430 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:48.851640 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:48.851605 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:53:58.046692 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:53:58.046597 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:54:11.148093 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:54:11.148036 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:54:19.941830 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:54:19.941791 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:54:29.546763 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:54:29.546728 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:54:41.539809 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:54:41.539773 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:54:51.936388 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:54:51.936329 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:55:00.115929 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:00.115901 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:55:00.117575 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:00.117551 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 19:55:00.238736 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:00.238695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:55:12.439501 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:12.439468 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:55:23.944977 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:23.944888 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:55:35.237683 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:35.237647 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:55:48.341874 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:55:48.341837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:56:01.340936 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:56:01.340903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:56:24.139266 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:56:24.139233 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:56:35.342991 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:56:35.342961 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:56:43.332780 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:56:43.332696 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:56:51.753547 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:56:51.753463 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:58:11.842171 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:58:11.842132 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:58:20.140562 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:58:20.140519 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:58:32.645320 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:58:32.645277 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:58:42.442313 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:58:42.442270 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:59:11.339328 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:11.339292 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:59:22.243778 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.243745 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 19:59:22.247537 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.247520 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.251703 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.251677 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5592633f\"/\"default-dockercfg-dh89l\"" Jul 05 19:59:22.251703 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.251691 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5592633f\"/\"kube-root-ca.crt\"" Jul 05 19:59:22.251897 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.251744 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5592633f\"/\"openshift-service-ca.crt\"" Jul 05 19:59:22.251897 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.251763 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5592633f\"/\"e2e-lifecycle-model-5592633f-kserve-self-signed-certs\"" Jul 05 19:59:22.257409 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.257389 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 19:59:22.358189 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358153 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.358366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358195 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb46n\" (UniqueName: \"kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.358366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358308 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.358366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358337 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.358366 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358358 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.358530 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.358375 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459258 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb46n\" (UniqueName: \"kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459353 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459427 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459643 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459792 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459771 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459862 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459836 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.459901 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.459872 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.461669 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.461640 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.461988 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.461973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.467350 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.467331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb46n\" (UniqueName: \"kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n\") pod \"e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.557708 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.557638 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:22.694585 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:22.694562 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 19:59:22.697183 ip-10-0-143-177 kubenswrapper[2581]: W0705 19:59:22.697148 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8db37c89_091f_459a_96a5_c9829574e67b.slice/crio-c4cef654ab4527b85e304ca41a11ad6657ae9e8b8e38e1f459fb88b3d23124a0 WatchSource:0}: Error finding container c4cef654ab4527b85e304ca41a11ad6657ae9e8b8e38e1f459fb88b3d23124a0: Status 404 returned error can't find the container with id c4cef654ab4527b85e304ca41a11ad6657ae9e8b8e38e1f459fb88b3d23124a0 Jul 05 19:59:23.313214 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:23.313178 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" event={"ID":"8db37c89-091f-459a-96a5-c9829574e67b","Type":"ContainerStarted","Data":"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa"} Jul 05 19:59:23.313214 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:23.313219 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" event={"ID":"8db37c89-091f-459a-96a5-c9829574e67b","Type":"ContainerStarted","Data":"c4cef654ab4527b85e304ca41a11ad6657ae9e8b8e38e1f459fb88b3d23124a0"} Jul 05 19:59:23.330347 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:23.330300 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" podStartSLOduration=1.330286277 podStartE2EDuration="1.330286277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:59:23.328750103 +0000 UTC m=+1763.784533534" watchObservedRunningTime="2026-07-05 19:59:23.330286277 +0000 UTC m=+1763.786069707" Jul 05 19:59:25.532891 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:25.532857 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 19:59:32.558481 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:32.558449 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:32.558481 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:32.558488 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:32.570986 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:32.570961 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:33.378657 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:33.378630 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 19:59:57.838901 ip-10-0-143-177 kubenswrapper[2581]: I0705 19:59:57.838824 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:00:00.154798 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:00.154771 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:00:00.157154 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:00.157131 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:00:11.251271 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:11.251232 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 20:00:11.536493 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:11.536403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:00:26.272931 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.272870 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" podUID="8db37c89-091f-459a-96a5-c9829574e67b" containerName="main" containerID="cri-o://b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa" gracePeriod=45 Jul 05 20:00:26.524525 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.524465 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 20:00:26.627915 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.627885 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628123 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.627935 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb46n\" (UniqueName: \"kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628123 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.627961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628123 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.627979 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628123 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628003 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628146 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm\") pod \"8db37c89-091f-459a-96a5-c9829574e67b\" (UID: \"8db37c89-091f-459a-96a5-c9829574e67b\") " Jul 05 20:00:26.628347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628244 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home" (OuterVolumeSpecName: "home") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:00:26.628347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628309 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:00:26.628347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628317 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:00:26.628547 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628527 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:26.628603 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628553 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:26.628603 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.628570 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:26.630427 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.630396 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:00:26.630539 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.630452 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm" (OuterVolumeSpecName: "dshm") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:00:26.630539 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.630469 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n" (OuterVolumeSpecName: "kube-api-access-gb46n") pod "8db37c89-091f-459a-96a5-c9829574e67b" (UID: "8db37c89-091f-459a-96a5-c9829574e67b"). InnerVolumeSpecName "kube-api-access-gb46n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:00:26.662867 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.662837 2581 generic.go:356] "Generic (PLEG): container finished" podID="8db37c89-091f-459a-96a5-c9829574e67b" containerID="b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa" exitCode=0 Jul 05 20:00:26.663013 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.662881 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" Jul 05 20:00:26.663013 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.662930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" event={"ID":"8db37c89-091f-459a-96a5-c9829574e67b","Type":"ContainerDied","Data":"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa"} Jul 05 20:00:26.663013 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.662968 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6" event={"ID":"8db37c89-091f-459a-96a5-c9829574e67b","Type":"ContainerDied","Data":"c4cef654ab4527b85e304ca41a11ad6657ae9e8b8e38e1f459fb88b3d23124a0"} Jul 05 20:00:26.663013 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.662986 2581 scope.go:117] "RemoveContainer" containerID="b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa" Jul 05 20:00:26.679429 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.679411 2581 scope.go:117] "RemoveContainer" containerID="b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa" Jul 05 20:00:26.679652 ip-10-0-143-177 kubenswrapper[2581]: E0705 20:00:26.679634 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa\": container with ID starting with b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa not found: ID does not exist" containerID="b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa" Jul 05 20:00:26.679696 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.679661 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa"} err="failed to get container status \"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa\": rpc error: code = NotFound desc = could not find container \"b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa\": container with ID starting with b8cc530964cb0ad05433848b0ad82e0cae221cde4393d688787de24b82abe1aa not found: ID does not exist" Jul 05 20:00:26.688402 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.688381 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 20:00:26.691898 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.691878 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5592633f/e2e-lifecycle-model-5592633f-kserve-795ff68c55-hzrk6"] Jul 05 20:00:26.729135 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.729113 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb46n\" (UniqueName: \"kubernetes.io/projected/8db37c89-091f-459a-96a5-c9829574e67b-kube-api-access-gb46n\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:26.729135 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.729132 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db37c89-091f-459a-96a5-c9829574e67b-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:26.729299 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:26.729142 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db37c89-091f-459a-96a5-c9829574e67b-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:00:28.060930 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:28.060881 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db37c89-091f-459a-96a5-c9829574e67b" path="/var/lib/kubelet/pods/8db37c89-091f-459a-96a5-c9829574e67b/volumes" Jul 05 20:00:50.939364 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:00:50.939329 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:01:03.550430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:01:03.550393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:01:12.543099 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:01:12.543062 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:01:58.741529 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:01:58.741494 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:02:09.739016 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:09.738976 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:02:23.650551 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.650518 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:02:23.650922 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.650910 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db37c89-091f-459a-96a5-c9829574e67b" containerName="main" Jul 05 20:02:23.650966 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.650924 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db37c89-091f-459a-96a5-c9829574e67b" containerName="main" Jul 05 20:02:23.651006 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.650987 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db37c89-091f-459a-96a5-c9829574e67b" containerName="main" Jul 05 20:02:23.654117 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.654100 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.658147 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.658088 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6354d0\"/\"kube-root-ca.crt\"" Jul 05 20:02:23.658147 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.658108 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6354d0\"/\"test-model-6354d0-kserve-self-signed-certs\"" Jul 05 20:02:23.658147 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.658090 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6354d0\"/\"default-dockercfg-fth96\"" Jul 05 20:02:23.658147 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.658092 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6354d0\"/\"openshift-service-ca.crt\"" Jul 05 20:02:23.661608 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.661586 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:02:23.782685 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782650 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.782886 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.782886 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttkk\" (UniqueName: \"kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.782886 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.783043 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782903 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.783043 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.782953 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883710 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883710 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883969 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttkk\" (UniqueName: \"kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883969 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883969 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.883969 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.883808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.884245 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.884217 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.884245 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.884237 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.884322 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.884297 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.886123 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.886100 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.886467 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.886448 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.891675 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.891650 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttkk\" (UniqueName: \"kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk\") pod \"test-model-6354d0-kserve-55866588bf-7srb9\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:23.963861 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:23.963823 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:24.109441 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:24.106903 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:02:24.304331 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:24.304247 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" event={"ID":"dad1a9e0-59b7-45a0-8d45-b891a70c8180","Type":"ContainerStarted","Data":"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94"} Jul 05 20:02:24.304331 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:24.304289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" event={"ID":"dad1a9e0-59b7-45a0-8d45-b891a70c8180","Type":"ContainerStarted","Data":"ed2b2b1da4d00b31b5b4176a94d0221d5d14fc301a91956466da192919d02cf5"} Jul 05 20:02:24.320686 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:24.320637 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" podStartSLOduration=1.320621858 podStartE2EDuration="1.320621858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:02:24.319186691 +0000 UTC m=+1944.774970121" watchObservedRunningTime="2026-07-05 20:02:24.320621858 +0000 UTC m=+1944.776405287" Jul 05 20:02:27.445749 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:27.445711 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:02:33.964533 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:33.964493 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:33.965006 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:33.964558 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:33.977148 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:33.977093 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:34.371253 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:34.371172 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:02:51.847783 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.847709 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:02:51.851188 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.851172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.855205 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.855180 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82d890\"/\"openshift-service-ca.crt\"" Jul 05 20:02:51.855351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.855252 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82d890\"/\"default-dockercfg-q8vkw\"" Jul 05 20:02:51.855351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.855180 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-82d890\"/\"test-model-82d890-kserve-self-signed-certs\"" Jul 05 20:02:51.855351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.855270 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-82d890\"/\"kube-root-ca.crt\"" Jul 05 20:02:51.859646 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.859625 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:02:51.935973 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.935939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.936261 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.935986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.936261 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.936127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtjk\" (UniqueName: \"kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.936261 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.936213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.936261 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.936241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:51.936414 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:51.936295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.036794 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtjk\" (UniqueName: \"kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037003 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037003 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037003 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036904 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037003 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036932 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037003 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.036974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037392 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.037367 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037460 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.037375 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.037460 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.037419 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.039296 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.039270 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.039565 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.039547 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.045877 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.045847 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtjk\" (UniqueName: \"kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk\") pod \"test-model-82d890-kserve-7579687b75-rz58z\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.162622 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.162582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:02:52.305711 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.305682 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:02:52.308246 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:02:52.308215 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67d81f1_95ed_4a9b_b059_c29f338879c2.slice/crio-14ddc07a94ed02e46e5f87dc2464c374ae9b64fa05e8bf1af80be573c575e816 WatchSource:0}: Error finding container 14ddc07a94ed02e46e5f87dc2464c374ae9b64fa05e8bf1af80be573c575e816: Status 404 returned error can't find the container with id 14ddc07a94ed02e46e5f87dc2464c374ae9b64fa05e8bf1af80be573c575e816 Jul 05 20:02:52.469764 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.469662 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" event={"ID":"a67d81f1-95ed-4a9b-b059-c29f338879c2","Type":"ContainerStarted","Data":"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b"} Jul 05 20:02:52.469764 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.469711 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" event={"ID":"a67d81f1-95ed-4a9b-b059-c29f338879c2","Type":"ContainerStarted","Data":"14ddc07a94ed02e46e5f87dc2464c374ae9b64fa05e8bf1af80be573c575e816"} Jul 05 20:02:52.486373 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:52.486320 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" podStartSLOduration=1.4863035789999999 podStartE2EDuration="1.486303579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:02:52.485634966 +0000 UTC m=+1972.941418395" watchObservedRunningTime="2026-07-05 20:02:52.486303579 +0000 UTC m=+1972.942087011" Jul 05 20:02:54.640196 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:02:54.640162 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:03:02.163583 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:02.163540 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:02.163583 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:02.163592 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:02.176341 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:02.176317 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:02.539980 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:02.539902 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:06.638602 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:06.638557 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:03:11.927214 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:11.927185 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:03:12.833695 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:12.833657 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:03:18.638358 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:18.638322 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:03:26.951248 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:26.951185 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" podUID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" containerName="main" containerID="cri-o://09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94" gracePeriod=45 Jul 05 20:03:27.199209 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.199182 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:03:27.279347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279262 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vttkk\" (UniqueName: \"kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279306 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279347 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279337 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279611 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279361 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279611 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279406 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279611 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279427 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm\") pod \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\" (UID: \"dad1a9e0-59b7-45a0-8d45-b891a70c8180\") " Jul 05 20:03:27.279749 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279655 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home" (OuterVolumeSpecName: "home") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:27.279749 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279686 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:27.279749 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279736 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:27.279868 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.279780 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.281727 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.281703 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk" (OuterVolumeSpecName: "kube-api-access-vttkk") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "kube-api-access-vttkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:03:27.281852 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.281737 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:03:27.281852 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.281767 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm" (OuterVolumeSpecName: "dshm") pod "dad1a9e0-59b7-45a0-8d45-b891a70c8180" (UID: "dad1a9e0-59b7-45a0-8d45-b891a70c8180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:27.380300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.380269 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttkk\" (UniqueName: \"kubernetes.io/projected/dad1a9e0-59b7-45a0-8d45-b891a70c8180-kube-api-access-vttkk\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.380300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.380296 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.380300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.380307 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.380515 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.380314 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.380515 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.380321 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad1a9e0-59b7-45a0-8d45-b891a70c8180-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:27.666198 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.666162 2581 generic.go:356] "Generic (PLEG): container finished" podID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" containerID="09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94" exitCode=0 Jul 05 20:03:27.666386 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.666216 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" Jul 05 20:03:27.666386 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.666251 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" event={"ID":"dad1a9e0-59b7-45a0-8d45-b891a70c8180","Type":"ContainerDied","Data":"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94"} Jul 05 20:03:27.666386 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.666299 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9" event={"ID":"dad1a9e0-59b7-45a0-8d45-b891a70c8180","Type":"ContainerDied","Data":"ed2b2b1da4d00b31b5b4176a94d0221d5d14fc301a91956466da192919d02cf5"} Jul 05 20:03:27.666386 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.666311 2581 scope.go:117] "RemoveContainer" containerID="09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94" Jul 05 20:03:27.676010 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.675992 2581 scope.go:117] "RemoveContainer" containerID="09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94" Jul 05 20:03:27.676294 ip-10-0-143-177 kubenswrapper[2581]: E0705 20:03:27.676277 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94\": container with ID starting with 09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94 not found: ID does not exist" containerID="09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94" Jul 05 20:03:27.676338 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.676303 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94"} err="failed to get container status \"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94\": rpc error: code = NotFound desc = could not find container \"09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94\": container with ID starting with 09ef1109830e998819a359754e5b0e7898408fb348991248bb65a093948bcc94 not found: ID does not exist" Jul 05 20:03:27.687974 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.687947 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:03:27.691076 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.691035 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6354d0/test-model-6354d0-kserve-55866588bf-7srb9"] Jul 05 20:03:27.854991 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:27.854948 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" podUID="a67d81f1-95ed-4a9b-b059-c29f338879c2" containerName="main" containerID="cri-o://ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b" gracePeriod=45 Jul 05 20:03:28.063865 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.063835 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" path="/var/lib/kubelet/pods/dad1a9e0-59b7-45a0-8d45-b891a70c8180/volumes" Jul 05 20:03:28.115774 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.115747 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:28.185777 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185745 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.185941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185798 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smtjk\" (UniqueName: \"kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.185941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185831 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.185941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185858 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.185941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185883 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.186173 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.185967 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home\") pod \"a67d81f1-95ed-4a9b-b059-c29f338879c2\" (UID: \"a67d81f1-95ed-4a9b-b059-c29f338879c2\") " Jul 05 20:03:28.186173 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.186087 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:28.186300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.186274 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.186340 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.186272 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:28.186340 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.186286 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home" (OuterVolumeSpecName: "home") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:28.188274 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.188253 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm" (OuterVolumeSpecName: "dshm") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:03:28.188395 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.188268 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk" (OuterVolumeSpecName: "kube-api-access-smtjk") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "kube-api-access-smtjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:03:28.188395 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.188298 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a67d81f1-95ed-4a9b-b059-c29f338879c2" (UID: "a67d81f1-95ed-4a9b-b059-c29f338879c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:03:28.286732 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.286655 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.286732 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.286680 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtjk\" (UniqueName: \"kubernetes.io/projected/a67d81f1-95ed-4a9b-b059-c29f338879c2-kube-api-access-smtjk\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.286732 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.286690 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a67d81f1-95ed-4a9b-b059-c29f338879c2-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.286732 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.286698 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.286732 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.286705 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a67d81f1-95ed-4a9b-b059-c29f338879c2-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:03:28.680792 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.680752 2581 generic.go:356] "Generic (PLEG): container finished" podID="a67d81f1-95ed-4a9b-b059-c29f338879c2" containerID="ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b" exitCode=0 Jul 05 20:03:28.680950 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.680804 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" Jul 05 20:03:28.680950 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.680839 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" event={"ID":"a67d81f1-95ed-4a9b-b059-c29f338879c2","Type":"ContainerDied","Data":"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b"} Jul 05 20:03:28.680950 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.680880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z" event={"ID":"a67d81f1-95ed-4a9b-b059-c29f338879c2","Type":"ContainerDied","Data":"14ddc07a94ed02e46e5f87dc2464c374ae9b64fa05e8bf1af80be573c575e816"} Jul 05 20:03:28.680950 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.680897 2581 scope.go:117] "RemoveContainer" containerID="ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b" Jul 05 20:03:28.691254 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.691240 2581 scope.go:117] "RemoveContainer" containerID="ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b" Jul 05 20:03:28.691494 ip-10-0-143-177 kubenswrapper[2581]: E0705 20:03:28.691467 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b\": container with ID starting with ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b not found: ID does not exist" containerID="ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b" Jul 05 20:03:28.691595 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.691495 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b"} err="failed to get container status \"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b\": rpc error: code = NotFound desc = could not find container \"ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b\": container with ID starting with ce69a41240bf54fe6ba76ac7a666946df30557dbbbad70bcc017e0adc6edf62b not found: ID does not exist" Jul 05 20:03:28.702853 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.702827 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:03:28.708383 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:28.708362 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-82d890/test-model-82d890-kserve-7579687b75-rz58z"] Jul 05 20:03:30.061178 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:30.061142 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67d81f1-95ed-4a9b-b059-c29f338879c2" path="/var/lib/kubelet/pods/a67d81f1-95ed-4a9b-b059-c29f338879c2/volumes" Jul 05 20:03:58.639816 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:03:58.639781 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:04:11.643599 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:04:11.643559 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:04:21.334473 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:04:21.334393 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:04:32.741313 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:04:32.741275 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:04:42.434561 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:04:42.434522 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:04:52.537912 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:04:52.537871 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:05:00.192115 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:00.192090 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:05:00.196414 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:00.196393 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:05:01.236406 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:01.236369 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:05:08.848920 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.848886 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:05:08.849326 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849274 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67d81f1-95ed-4a9b-b059-c29f338879c2" containerName="main" Jul 05 20:05:08.849326 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849286 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67d81f1-95ed-4a9b-b059-c29f338879c2" containerName="main" Jul 05 20:05:08.849326 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849299 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" containerName="main" Jul 05 20:05:08.849326 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849304 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" containerName="main" Jul 05 20:05:08.849451 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849361 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad1a9e0-59b7-45a0-8d45-b891a70c8180" containerName="main" Jul 05 20:05:08.849451 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.849372 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67d81f1-95ed-4a9b-b059-c29f338879c2" containerName="main" Jul 05 20:05:08.852235 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.852220 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.856017 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.855989 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"openshift-service-ca.crt\"" Jul 05 20:05:08.856017 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.856015 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"kube-root-ca.crt\"" Jul 05 20:05:08.856216 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.855994 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"auth-test-model-ec8ab7-kserve-self-signed-certs\"" Jul 05 20:05:08.856216 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.855991 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"default-dockercfg-g4gcj\"" Jul 05 20:05:08.859718 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.859696 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:05:08.888518 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888483 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.888674 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888543 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.888674 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888570 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.888674 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj59k\" (UniqueName: \"kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.888674 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.888674 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.888660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990251 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990219 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990339 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj59k\" (UniqueName: \"kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990380 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990430 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990735 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990714 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.990973 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990946 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.991165 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.990973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.993300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.993270 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.993642 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.993621 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:08.999629 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:08.999607 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj59k\" (UniqueName: \"kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k\") pod \"auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:09.161840 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:09.161805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:09.300780 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:09.300756 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:05:09.302383 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:05:09.302344 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ef36004_14e0_42a3_9241_d2ed0e3b4705.slice/crio-f699fdfe5fa36766fb91cc0cd8e155e565332419f2e86fa4ee25b0ab704823c2 WatchSource:0}: Error finding container f699fdfe5fa36766fb91cc0cd8e155e565332419f2e86fa4ee25b0ab704823c2: Status 404 returned error can't find the container with id f699fdfe5fa36766fb91cc0cd8e155e565332419f2e86fa4ee25b0ab704823c2 Jul 05 20:05:10.232011 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:10.231977 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" event={"ID":"6ef36004-14e0-42a3-9241-d2ed0e3b4705","Type":"ContainerStarted","Data":"0828c3be8a76ae17a0be322a065c52d6a952b109b3c46a30a318dcc2c0111eef"} Jul 05 20:05:10.232390 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:10.232023 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" event={"ID":"6ef36004-14e0-42a3-9241-d2ed0e3b4705","Type":"ContainerStarted","Data":"f699fdfe5fa36766fb91cc0cd8e155e565332419f2e86fa4ee25b0ab704823c2"} Jul 05 20:05:10.249383 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:10.249335 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" podStartSLOduration=2.249317906 podStartE2EDuration="2.249317906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:05:10.247283675 +0000 UTC m=+2110.703067102" watchObservedRunningTime="2026-07-05 20:05:10.249317906 +0000 UTC m=+2110.705101375" Jul 05 20:05:12.445512 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:12.445479 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:05:19.162331 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:19.162297 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:19.162742 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:19.162344 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:19.174505 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:19.174477 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:19.288794 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:19.288769 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:05:38.149523 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:05:38.149486 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:06:07.841207 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:06:07.841167 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:06:25.143436 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:06:25.143398 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:06:41.147636 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:06:41.147596 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:06:59.639916 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:06:59.639880 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:07:12.841146 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:12.841105 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:07:29.041176 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:29.041094 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:07:43.028893 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:43.028857 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:07:46.138586 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:46.138550 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:07:56.732045 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.732017 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:07:56.735596 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.735575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.738292 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.738270 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"sub-test-model-ec8ab7-kserve-self-signed-certs\"" Jul 05 20:07:56.744282 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.744251 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:07:56.833731 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833705 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.833867 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.833867 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklmk\" (UniqueName: \"kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.833867 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833838 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.833989 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.833989 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.833956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935112 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935086 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935230 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935121 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935230 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935158 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935230 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935190 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935230 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklmk\" (UniqueName: \"kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935428 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935254 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935505 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935484 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935570 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935555 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.935647 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.935628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.937580 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.937553 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.937889 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.937871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:56.944354 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:56.944337 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklmk\" (UniqueName: \"kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk\") pod \"sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:57.046934 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:57.046851 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:07:57.188467 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:57.188434 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:07:57.189786 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:07:57.189748 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ea0a65d_e1e9_4029_aa03_39cf402b06ac.slice/crio-bebac69055c24d24bd3101f62038a7196299b37ce11a756c52eea7658dfde5d0 WatchSource:0}: Error finding container bebac69055c24d24bd3101f62038a7196299b37ce11a756c52eea7658dfde5d0: Status 404 returned error can't find the container with id bebac69055c24d24bd3101f62038a7196299b37ce11a756c52eea7658dfde5d0 Jul 05 20:07:58.049647 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.049591 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" podUID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" containerName="main" containerID="cri-o://0828c3be8a76ae17a0be322a065c52d6a952b109b3c46a30a318dcc2c0111eef" gracePeriod=45 Jul 05 20:07:58.174979 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.174946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" event={"ID":"3ea0a65d-e1e9-4029-aa03-39cf402b06ac","Type":"ContainerStarted","Data":"1f6c9b4cc1cbae498b4f6123c69cf3c978927c5d38764718817de0fe953e4f81"} Jul 05 20:07:58.175129 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.174994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" event={"ID":"3ea0a65d-e1e9-4029-aa03-39cf402b06ac","Type":"ContainerStarted","Data":"bebac69055c24d24bd3101f62038a7196299b37ce11a756c52eea7658dfde5d0"} Jul 05 20:07:58.176525 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.176500 2581 generic.go:356] "Generic (PLEG): container finished" podID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" containerID="0828c3be8a76ae17a0be322a065c52d6a952b109b3c46a30a318dcc2c0111eef" exitCode=0 Jul 05 20:07:58.176633 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.176553 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" event={"ID":"6ef36004-14e0-42a3-9241-d2ed0e3b4705","Type":"ContainerDied","Data":"0828c3be8a76ae17a0be322a065c52d6a952b109b3c46a30a318dcc2c0111eef"} Jul 05 20:07:58.195225 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.195179 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" podStartSLOduration=2.195162645 podStartE2EDuration="2.195162645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:07:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:07:58.192496719 +0000 UTC m=+2278.648280150" watchObservedRunningTime="2026-07-05 20:07:58.195162645 +0000 UTC m=+2278.650946076" Jul 05 20:07:58.308965 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.308901 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:07:58.450289 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450260 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj59k\" (UniqueName: \"kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450448 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450296 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450448 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450343 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450448 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450370 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450595 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450465 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450595 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450504 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm\") pod \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\" (UID: \"6ef36004-14e0-42a3-9241-d2ed0e3b4705\") " Jul 05 20:07:58.450595 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450580 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:58.450751 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450600 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:58.450815 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450742 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home" (OuterVolumeSpecName: "home") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:58.450883 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450863 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:58.450934 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450889 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:58.450934 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.450903 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:58.452654 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.452628 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm" (OuterVolumeSpecName: "dshm") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:58.452773 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.452659 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:07:58.452773 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.452742 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k" (OuterVolumeSpecName: "kube-api-access-tj59k") pod "6ef36004-14e0-42a3-9241-d2ed0e3b4705" (UID: "6ef36004-14e0-42a3-9241-d2ed0e3b4705"). InnerVolumeSpecName "kube-api-access-tj59k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:07:58.551672 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.551631 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef36004-14e0-42a3-9241-d2ed0e3b4705-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:58.551672 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.551668 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj59k\" (UniqueName: \"kubernetes.io/projected/6ef36004-14e0-42a3-9241-d2ed0e3b4705-kube-api-access-tj59k\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:58.551672 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:58.551680 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef36004-14e0-42a3-9241-d2ed0e3b4705-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.182931 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:59.182895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" event={"ID":"6ef36004-14e0-42a3-9241-d2ed0e3b4705","Type":"ContainerDied","Data":"f699fdfe5fa36766fb91cc0cd8e155e565332419f2e86fa4ee25b0ab704823c2"} Jul 05 20:07:59.182931 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:59.182916 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n" Jul 05 20:07:59.183459 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:59.182949 2581 scope.go:117] "RemoveContainer" containerID="0828c3be8a76ae17a0be322a065c52d6a952b109b3c46a30a318dcc2c0111eef" Jul 05 20:07:59.203978 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:59.203955 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:07:59.208456 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:07:59.208436 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/auth-test-model-ec8ab7-kserve-64875bfdff-l4c2n"] Jul 05 20:08:00.061942 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:00.061911 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" path="/var/lib/kubelet/pods/6ef36004-14e0-42a3-9241-d2ed0e3b4705/volumes" Jul 05 20:08:01.639961 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:01.639931 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:08:07.047303 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:07.047262 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:08:07.047303 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:07.047316 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:08:07.059923 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:07.059901 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:08:07.248117 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:07.248091 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:08:25.146890 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.146852 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:08:25.147429 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.147407 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" containerName="main" Jul 05 20:08:25.147473 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.147434 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" containerName="main" Jul 05 20:08:25.147603 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.147591 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef36004-14e0-42a3-9241-d2ed0e3b4705" containerName="main" Jul 05 20:08:25.150528 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.150512 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.153146 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.153123 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"openshift-service-ca.crt\"" Jul 05 20:08:25.154446 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.154430 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"kube-root-ca.crt\"" Jul 05 20:08:25.154544 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.154444 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"sub-test-model-c11b28-kserve-self-signed-certs\"" Jul 05 20:08:25.154544 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.154521 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"default-dockercfg-tx5b9\"" Jul 05 20:08:25.159477 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.159456 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:08:25.189217 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.189309 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zz9f\" (UniqueName: \"kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.189309 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.189389 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189334 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.189389 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189384 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.189458 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.189403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.290734 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zz9f\" (UniqueName: \"kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.290734 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290737 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.290941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.290941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.290941 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290867 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.291138 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.290940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.291341 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.291317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.291425 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.291347 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.291425 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.291409 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.293214 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.293194 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.293758 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.293741 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.299344 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.299320 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zz9f\" (UniqueName: \"kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f\") pod \"sub-test-model-c11b28-kserve-66cf99c765-4t6l4\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.460232 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.460204 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:25.596363 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:25.596336 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:08:25.597790 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:08:25.597755 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b8f9343_3e55_408e_89ce_1a2da1eae3de.slice/crio-c3b6927a2263c4a7016ba6b8c2992b042466f3f25ae4f1b4313fe3ddbeec0685 WatchSource:0}: Error finding container c3b6927a2263c4a7016ba6b8c2992b042466f3f25ae4f1b4313fe3ddbeec0685: Status 404 returned error can't find the container with id c3b6927a2263c4a7016ba6b8c2992b042466f3f25ae4f1b4313fe3ddbeec0685 Jul 05 20:08:26.342110 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:26.342074 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" event={"ID":"5b8f9343-3e55-408e-89ce-1a2da1eae3de","Type":"ContainerStarted","Data":"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98"} Jul 05 20:08:26.342110 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:26.342114 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" event={"ID":"5b8f9343-3e55-408e-89ce-1a2da1eae3de","Type":"ContainerStarted","Data":"c3b6927a2263c4a7016ba6b8c2992b042466f3f25ae4f1b4313fe3ddbeec0685"} Jul 05 20:08:26.359881 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:26.359833 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" podStartSLOduration=1.359819185 podStartE2EDuration="1.359819185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:08:26.356456857 +0000 UTC m=+2306.812240287" watchObservedRunningTime="2026-07-05 20:08:26.359819185 +0000 UTC m=+2306.815602615" Jul 05 20:08:29.546183 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:29.546150 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:08:35.460645 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:35.460607 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:35.461018 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:35.460699 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:35.473008 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:35.472967 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:36.406470 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:36.406443 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:08:45.743269 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:08:45.743230 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:09:02.420751 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:02.420716 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:09:03.245212 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:03.245178 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:09:06.431660 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:06.431626 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:09:17.441125 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.441081 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" podUID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" containerName="main" containerID="cri-o://1f6c9b4cc1cbae498b4f6123c69cf3c978927c5d38764718817de0fe953e4f81" gracePeriod=45 Jul 05 20:09:17.625385 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.625356 2581 generic.go:356] "Generic (PLEG): container finished" podID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" containerID="1f6c9b4cc1cbae498b4f6123c69cf3c978927c5d38764718817de0fe953e4f81" exitCode=0 Jul 05 20:09:17.625496 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.625422 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" event={"ID":"3ea0a65d-e1e9-4029-aa03-39cf402b06ac","Type":"ContainerDied","Data":"1f6c9b4cc1cbae498b4f6123c69cf3c978927c5d38764718817de0fe953e4f81"} Jul 05 20:09:17.697383 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.697328 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:09:17.745879 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.745853 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746024 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.745916 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lklmk\" (UniqueName: \"kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746024 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.745939 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746024 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.745955 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746024 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.745975 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746261 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746087 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache\") pod \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\" (UID: \"3ea0a65d-e1e9-4029-aa03-39cf402b06ac\") " Jul 05 20:09:17.746321 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746253 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home" (OuterVolumeSpecName: "home") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:17.746321 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746267 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:17.746321 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746280 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:17.746498 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746480 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:17.746712 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746524 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:17.746712 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.746540 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:17.748173 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.748145 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm" (OuterVolumeSpecName: "dshm") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:17.748266 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.748231 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:09:17.748266 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.748239 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk" (OuterVolumeSpecName: "kube-api-access-lklmk") pod "3ea0a65d-e1e9-4029-aa03-39cf402b06ac" (UID: "3ea0a65d-e1e9-4029-aa03-39cf402b06ac"). InnerVolumeSpecName "kube-api-access-lklmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:09:17.847322 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.847298 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:17.847322 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.847317 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:17.847322 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:17.847326 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklmk\" (UniqueName: \"kubernetes.io/projected/3ea0a65d-e1e9-4029-aa03-39cf402b06ac-kube-api-access-lklmk\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.265460 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.265421 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" podUID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" containerName="main" containerID="cri-o://c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98" gracePeriod=45 Jul 05 20:09:18.522287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.522263 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:09:18.554478 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554451 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554616 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554488 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554616 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554506 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554616 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554546 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554616 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554588 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554616 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554607 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zz9f\" (UniqueName: \"kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f\") pod \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\" (UID: \"5b8f9343-3e55-408e-89ce-1a2da1eae3de\") " Jul 05 20:09:18.554884 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554824 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:18.554884 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554836 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:18.554965 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.554952 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home" (OuterVolumeSpecName: "home") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:18.556761 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.556739 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:09:18.556869 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.556855 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f" (OuterVolumeSpecName: "kube-api-access-5zz9f") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "kube-api-access-5zz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:09:18.556959 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.556944 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm" (OuterVolumeSpecName: "dshm") pod "5b8f9343-3e55-408e-89ce-1a2da1eae3de" (UID: "5b8f9343-3e55-408e-89ce-1a2da1eae3de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:09:18.631679 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.631646 2581 generic.go:356] "Generic (PLEG): container finished" podID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" containerID="c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98" exitCode=0 Jul 05 20:09:18.631842 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.631701 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" Jul 05 20:09:18.631842 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.631733 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" event={"ID":"5b8f9343-3e55-408e-89ce-1a2da1eae3de","Type":"ContainerDied","Data":"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98"} Jul 05 20:09:18.631842 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.631776 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4" event={"ID":"5b8f9343-3e55-408e-89ce-1a2da1eae3de","Type":"ContainerDied","Data":"c3b6927a2263c4a7016ba6b8c2992b042466f3f25ae4f1b4313fe3ddbeec0685"} Jul 05 20:09:18.631842 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.631797 2581 scope.go:117] "RemoveContainer" containerID="c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98" Jul 05 20:09:18.633264 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.633246 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" Jul 05 20:09:18.633370 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.633249 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w" event={"ID":"3ea0a65d-e1e9-4029-aa03-39cf402b06ac","Type":"ContainerDied","Data":"bebac69055c24d24bd3101f62038a7196299b37ce11a756c52eea7658dfde5d0"} Jul 05 20:09:18.640943 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.640928 2581 scope.go:117] "RemoveContainer" containerID="c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98" Jul 05 20:09:18.641242 ip-10-0-143-177 kubenswrapper[2581]: E0705 20:09:18.641223 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98\": container with ID starting with c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98 not found: ID does not exist" containerID="c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98" Jul 05 20:09:18.641309 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.641249 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98"} err="failed to get container status \"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98\": rpc error: code = NotFound desc = could not find container \"c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98\": container with ID starting with c4ee0b2e755c47cb05cc4909d7449aed38ffe5aec8b7971d4005ae06c0dc5a98 not found: ID does not exist" Jul 05 20:09:18.641309 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.641264 2581 scope.go:117] "RemoveContainer" containerID="1f6c9b4cc1cbae498b4f6123c69cf3c978927c5d38764718817de0fe953e4f81" Jul 05 20:09:18.649212 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.649192 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:09:18.651696 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.651678 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/sub-test-model-ec8ab7-kserve-79f7c8c87f-pj62w"] Jul 05 20:09:18.655437 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655415 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.655437 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655435 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zz9f\" (UniqueName: \"kubernetes.io/projected/5b8f9343-3e55-408e-89ce-1a2da1eae3de-kube-api-access-5zz9f\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.655559 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655445 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.655559 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655454 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.655559 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655461 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.655559 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.655468 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b8f9343-3e55-408e-89ce-1a2da1eae3de-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:09:18.660396 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.660377 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:09:18.663667 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:18.663649 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/sub-test-model-c11b28-kserve-66cf99c765-4t6l4"] Jul 05 20:09:20.062475 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:20.062444 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" path="/var/lib/kubelet/pods/3ea0a65d-e1e9-4029-aa03-39cf402b06ac/volumes" Jul 05 20:09:20.062826 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:20.062810 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" path="/var/lib/kubelet/pods/5b8f9343-3e55-408e-89ce-1a2da1eae3de/volumes" Jul 05 20:09:23.442389 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:23.442358 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:09:38.534684 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.534651 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:09:38.535310 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535234 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" containerName="main" Jul 05 20:09:38.535310 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535253 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" containerName="main" Jul 05 20:09:38.535310 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535278 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" containerName="main" Jul 05 20:09:38.535310 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535287 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" containerName="main" Jul 05 20:09:38.535552 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535379 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea0a65d-e1e9-4029-aa03-39cf402b06ac" containerName="main" Jul 05 20:09:38.535552 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.535393 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8f9343-3e55-408e-89ce-1a2da1eae3de" containerName="main" Jul 05 20:09:38.538613 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.538590 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.542498 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.542474 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"openshift-service-ca.crt\"" Jul 05 20:09:38.542597 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.542474 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"rate-test-model-c11b28-kserve-self-signed-certs\"" Jul 05 20:09:38.542597 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.542478 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"default-dockercfg-tx5b9\"" Jul 05 20:09:38.542597 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.542515 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"kube-root-ca.crt\"" Jul 05 20:09:38.548278 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.548254 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:09:38.631193 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631148 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.631369 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.631369 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.631369 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.631369 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631361 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.631516 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.631382 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7r7l\" (UniqueName: \"kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732369 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732553 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732374 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732553 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732501 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732553 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732534 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7r7l\" (UniqueName: \"kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732725 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732725 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732820 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732820 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.732991 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.732973 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.734828 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.734799 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.735097 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.735082 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.740067 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.740037 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7r7l\" (UniqueName: \"kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l\") pod \"rate-test-model-c11b28-kserve-89555d8f9-5n25z\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.849282 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.849198 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:38.983954 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:38.983932 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:09:38.985838 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:09:38.985812 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd06d5f6c_a010_4b0f_b1ec_8ab2f910dc35.slice/crio-812dd9f96ffd1f471ce64f632095a98c15fa434eef5ffb2fdd57e8dcc637c286 WatchSource:0}: Error finding container 812dd9f96ffd1f471ce64f632095a98c15fa434eef5ffb2fdd57e8dcc637c286: Status 404 returned error can't find the container with id 812dd9f96ffd1f471ce64f632095a98c15fa434eef5ffb2fdd57e8dcc637c286 Jul 05 20:09:39.753627 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:39.753596 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" event={"ID":"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35","Type":"ContainerStarted","Data":"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097"} Jul 05 20:09:39.754105 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:39.753634 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" event={"ID":"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35","Type":"ContainerStarted","Data":"812dd9f96ffd1f471ce64f632095a98c15fa434eef5ffb2fdd57e8dcc637c286"} Jul 05 20:09:39.772117 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:39.772071 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" podStartSLOduration=1.7720412620000001 podStartE2EDuration="1.772041262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:09:39.768811592 +0000 UTC m=+2380.224595044" watchObservedRunningTime="2026-07-05 20:09:39.772041262 +0000 UTC m=+2380.227824690" Jul 05 20:09:42.438719 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:42.438684 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:09:48.849577 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:48.849539 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:48.849976 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:48.849853 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:48.861983 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:48.861961 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:09:49.821309 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:09:49.821282 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:10:00.230456 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:00.230426 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:10:00.234931 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:00.234908 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:10:16.138579 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:16.138537 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:10:29.870533 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:29.870460 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:10:38.727627 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:38.727590 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:10:43.537541 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:43.537509 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:10:53.748288 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:53.748249 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" podUID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" containerName="main" containerID="cri-o://1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097" gracePeriod=45 Jul 05 20:10:54.008857 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.008799 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:10:54.130272 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130241 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130463 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130282 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7r7l\" (UniqueName: \"kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130463 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130316 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130463 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130360 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130463 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130391 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130463 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130434 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm\") pod \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\" (UID: \"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35\") " Jul 05 20:10:54.130730 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130641 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home" (OuterVolumeSpecName: "home") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:54.130730 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130654 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache" (OuterVolumeSpecName: "model-cache") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:54.130730 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130682 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:54.130885 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.130867 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-home\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:54.132541 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.132516 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l" (OuterVolumeSpecName: "kube-api-access-k7r7l") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "kube-api-access-k7r7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:10:54.133034 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.133012 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm" (OuterVolumeSpecName: "dshm") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:54.133149 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.133029 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" (UID: "d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:10:54.156907 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.156884 2581 generic.go:356] "Generic (PLEG): container finished" podID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" containerID="1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097" exitCode=0 Jul 05 20:10:54.157028 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.156924 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" Jul 05 20:10:54.157028 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.156918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" event={"ID":"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35","Type":"ContainerDied","Data":"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097"} Jul 05 20:10:54.157028 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.156974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z" event={"ID":"d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35","Type":"ContainerDied","Data":"812dd9f96ffd1f471ce64f632095a98c15fa434eef5ffb2fdd57e8dcc637c286"} Jul 05 20:10:54.157028 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.156998 2581 scope.go:117] "RemoveContainer" containerID="1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097" Jul 05 20:10:54.169000 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.168972 2581 scope.go:117] "RemoveContainer" containerID="1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097" Jul 05 20:10:54.169473 ip-10-0-143-177 kubenswrapper[2581]: E0705 20:10:54.169447 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097\": container with ID starting with 1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097 not found: ID does not exist" containerID="1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097" Jul 05 20:10:54.169561 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.169484 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097"} err="failed to get container status \"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097\": rpc error: code = NotFound desc = could not find container \"1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097\": container with ID starting with 1e8a023a451ed49063abc2e293a11acb62b268429872e91f76b0727e45043097 not found: ID does not exist" Jul 05 20:10:54.179351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.179327 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:10:54.182974 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.182956 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/rate-test-model-c11b28-kserve-89555d8f9-5n25z"] Jul 05 20:10:54.232235 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.232210 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-model-cache\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:54.232235 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.232231 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tmp-dir\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:54.232235 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.232239 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-dshm\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:54.232420 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.232246 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-tls-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:54.232420 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:54.232255 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7r7l\" (UniqueName: \"kubernetes.io/projected/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35-kube-api-access-k7r7l\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Jul 05 20:10:56.060963 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:10:56.060933 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" path="/var/lib/kubelet/pods/d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35/volumes" Jul 05 20:11:21.143794 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:21.143757 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-85wpn"] Jul 05 20:11:29.056785 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.056750 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lfkxp_9963cb83-7685-498e-a64d-5a48e4328664/manager/0.log" Jul 05 20:11:29.168574 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.168546 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rzk4c_2b5a1198-388f-42dd-b505-59ffc6c818d5/manager/0.log" Jul 05 20:11:29.724067 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.724032 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mmrr2_8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed/server/0.log" Jul 05 20:11:29.832259 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.832232 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2kbrx_cfb0bcc4-f937-4901-bba7-6d76871b3a73/manager/0.log" Jul 05 20:11:29.946277 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.946250 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-648kt_2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6/manager/0.log" Jul 05 20:11:29.953036 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:29.953016 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-648kt_2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6/copy-manifests/0.log" Jul 05 20:11:31.072811 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.072777 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/util/0.log" Jul 05 20:11:31.078667 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.078643 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/pull/0.log" Jul 05 20:11:31.084238 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.084209 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/extract/0.log" Jul 05 20:11:31.200872 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.200843 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/extract/0.log" Jul 05 20:11:31.206524 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.206509 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/util/0.log" Jul 05 20:11:31.211876 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.211859 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/pull/0.log" Jul 05 20:11:31.325507 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.325427 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/extract/0.log" Jul 05 20:11:31.331427 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.331404 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/util/0.log" Jul 05 20:11:31.337384 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.337368 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/pull/0.log" Jul 05 20:11:31.446967 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.446935 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/util/0.log" Jul 05 20:11:31.452485 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.452458 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/pull/0.log" Jul 05 20:11:31.457892 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.457874 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/extract/0.log" Jul 05 20:11:31.683668 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.683643 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9xdp5_bcf5f16a-94bb-4a20-b73a-39eb068ecdf2/manager/0.log" Jul 05 20:11:31.786446 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.786417 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8bqjp_4259dc9b-4833-4c2e-99ef-f4579ed3ebb3/manager/0.log" Jul 05 20:11:31.896969 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:31.896942 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bbnj8_efbfc0c7-6728-4e44-8e8e-4331657ccf06/kuadrant-console-plugin/0.log" Jul 05 20:11:32.004646 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:32.004576 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jmn84_7f962fcb-6c4a-441a-844e-a8b57dfbd1d4/registry-server/0.log" Jul 05 20:11:32.119605 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:32.119577 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2sv44_ee2f7887-7aca-4045-a1b1-e47b181ec56a/manager/0.log" Jul 05 20:11:32.236549 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:32.236526 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-85wpn_dea16eac-389d-4e50-9bf5-f3bedb634720/limitador/0.log" Jul 05 20:11:32.356196 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:32.356124 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dclxh_b63306a6-fd48-4fa2-9ea8-c2970c818d7d/manager/0.log" Jul 05 20:11:32.946934 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:32.946906 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-5vhjg_ba0f13ae-ef6b-4aec-9489-9636efa3a0b6/kube-auth-proxy/0.log" Jul 05 20:11:33.424876 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:33.424848 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t26cp_8ca415f0-3600-4e50-8c56-72135202d508/payload-processing/0.log" Jul 05 20:11:34.204300 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:34.204275 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr7njb_d7439118-a02c-4c16-b210-39ba14a46266/main/0.log" Jul 05 20:11:34.422225 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:34.422192 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-m9g2b_d0879cc8-a2cb-4212-8dbe-b790cc0f1998/main/0.log" Jul 05 20:11:38.356245 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.356217 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-27wzv/must-gather-wtssx"] Jul 05 20:11:38.356678 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.356662 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" containerName="main" Jul 05 20:11:38.356728 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.356680 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" containerName="main" Jul 05 20:11:38.356763 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.356752 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06d5f6c-a010-4b0f-b1ec-8ab2f910dc35" containerName="main" Jul 05 20:11:38.359891 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.359876 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.362473 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.362445 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27wzv\"/\"kube-root-ca.crt\"" Jul 05 20:11:38.363621 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.363606 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27wzv\"/\"openshift-service-ca.crt\"" Jul 05 20:11:38.363683 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.363638 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-27wzv\"/\"default-dockercfg-62j8p\"" Jul 05 20:11:38.368502 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.368480 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/must-gather-wtssx"] Jul 05 20:11:38.421339 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.421315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f916fb-b293-4214-b94b-24c01b12907d-must-gather-output\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.421477 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.421381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmlp\" (UniqueName: \"kubernetes.io/projected/f9f916fb-b293-4214-b94b-24c01b12907d-kube-api-access-lcmlp\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.522458 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.522428 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmlp\" (UniqueName: \"kubernetes.io/projected/f9f916fb-b293-4214-b94b-24c01b12907d-kube-api-access-lcmlp\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.522590 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.522492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f916fb-b293-4214-b94b-24c01b12907d-must-gather-output\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.522805 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.522789 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f9f916fb-b293-4214-b94b-24c01b12907d-must-gather-output\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.530164 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.530138 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmlp\" (UniqueName: \"kubernetes.io/projected/f9f916fb-b293-4214-b94b-24c01b12907d-kube-api-access-lcmlp\") pod \"must-gather-wtssx\" (UID: \"f9f916fb-b293-4214-b94b-24c01b12907d\") " pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.671102 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.671081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/must-gather-wtssx" Jul 05 20:11:38.811768 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.811743 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/must-gather-wtssx"] Jul 05 20:11:38.814462 ip-10-0-143-177 kubenswrapper[2581]: W0705 20:11:38.814436 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9f916fb_b293_4214_b94b_24c01b12907d.slice/crio-fde1c50f721893c47b449f5838fe75b6ec66c24017e01c226c45fae4cfddc41e WatchSource:0}: Error finding container fde1c50f721893c47b449f5838fe75b6ec66c24017e01c226c45fae4cfddc41e: Status 404 returned error can't find the container with id fde1c50f721893c47b449f5838fe75b6ec66c24017e01c226c45fae4cfddc41e Jul 05 20:11:38.816333 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:38.816317 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 20:11:39.398663 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:39.398628 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/must-gather-wtssx" event={"ID":"f9f916fb-b293-4214-b94b-24c01b12907d","Type":"ContainerStarted","Data":"fde1c50f721893c47b449f5838fe75b6ec66c24017e01c226c45fae4cfddc41e"} Jul 05 20:11:40.408869 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:40.408831 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/must-gather-wtssx" event={"ID":"f9f916fb-b293-4214-b94b-24c01b12907d","Type":"ContainerStarted","Data":"7bfdaab8099725b34834a7159486176f364371dd06e101bf0649cf96ad54995a"} Jul 05 20:11:40.409474 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:40.408884 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/must-gather-wtssx" event={"ID":"f9f916fb-b293-4214-b94b-24c01b12907d","Type":"ContainerStarted","Data":"705e5be86177d44bb0f0bebd07b124703099102c8bf0e26f8a8b13431def6d79"} Jul 05 20:11:40.427221 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:40.427160 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-27wzv/must-gather-wtssx" podStartSLOduration=1.660063496 podStartE2EDuration="2.427137765s" totalImagesPullingTime="767.074269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:11:40.423985655 +0000 UTC m=+2500.879769086" watchObservedRunningTime="2026-07-05 20:11:40.427137765 +0000 UTC m=+2500.882921195" Jul 05 20:11:41.039739 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:41.039702 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6srrh_e3003f33-196f-47f9-ad7b-55f6f99c48eb/global-pull-secret-syncer/0.log" Jul 05 20:11:41.227675 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:41.227636 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vhv6b_b49904b6-fdf6-47d2-8235-bd1927aff2f9/konnectivity-agent/0.log" Jul 05 20:11:41.290107 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:41.289972 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-177.ec2.internal_bd4ad6f0c2d4023e232d083fe9e47834/haproxy/0.log" Jul 05 20:11:45.495681 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.495641 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/extract/0.log" Jul 05 20:11:45.524790 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.524761 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/util/0.log" Jul 05 20:11:45.546770 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.546741 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75978zbj_a531734b-ba20-4e1f-9ba2-ee74704d1be3/pull/0.log" Jul 05 20:11:45.573675 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.573568 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/extract/0.log" Jul 05 20:11:45.595740 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.595705 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/util/0.log" Jul 05 20:11:45.617744 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.617700 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0std6m_118799c7-f94c-4ad4-9e63-0762113682fb/pull/0.log" Jul 05 20:11:45.648134 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.648095 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/extract/0.log" Jul 05 20:11:45.671180 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.671148 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/util/0.log" Jul 05 20:11:45.694990 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.694943 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739tcdv_67afeb7a-3561-4e31-af88-7acb57ba96de/pull/0.log" Jul 05 20:11:45.724824 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.724793 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/extract/0.log" Jul 05 20:11:45.746351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.746268 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/util/0.log" Jul 05 20:11:45.769137 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:45.769073 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1tpcdr_7aea3cd2-e33d-4348-b61c-3854195745e3/pull/0.log" Jul 05 20:11:46.162423 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.162390 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9xdp5_bcf5f16a-94bb-4a20-b73a-39eb068ecdf2/manager/0.log" Jul 05 20:11:46.185920 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.185883 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8bqjp_4259dc9b-4833-4c2e-99ef-f4579ed3ebb3/manager/0.log" Jul 05 20:11:46.210479 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.210455 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bbnj8_efbfc0c7-6728-4e44-8e8e-4331657ccf06/kuadrant-console-plugin/0.log" Jul 05 20:11:46.255169 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.255136 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jmn84_7f962fcb-6c4a-441a-844e-a8b57dfbd1d4/registry-server/0.log" Jul 05 20:11:46.389734 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.389644 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2sv44_ee2f7887-7aca-4045-a1b1-e47b181ec56a/manager/0.log" Jul 05 20:11:46.412825 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.412744 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-85wpn_dea16eac-389d-4e50-9bf5-f3bedb634720/limitador/0.log" Jul 05 20:11:46.486654 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:46.486612 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dclxh_b63306a6-fd48-4fa2-9ea8-c2970c818d7d/manager/0.log" Jul 05 20:11:48.077917 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.077881 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/alertmanager/0.log" Jul 05 20:11:48.106686 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.106650 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/config-reloader/0.log" Jul 05 20:11:48.124351 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.124318 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/kube-rbac-proxy-web/0.log" Jul 05 20:11:48.145509 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.145480 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/kube-rbac-proxy/0.log" Jul 05 20:11:48.171344 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.171142 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/kube-rbac-proxy-metric/0.log" Jul 05 20:11:48.196883 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.196857 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/prom-label-proxy/0.log" Jul 05 20:11:48.227537 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.227422 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0289d-3382-497d-a542-fc02b87d7096/init-config-reloader/0.log" Jul 05 20:11:48.370236 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.370155 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-558b58c774-6wrbd_8bf99b85-5a74-4874-9776-95b090766765/metrics-server/0.log" Jul 05 20:11:48.394507 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.394432 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-6xnpc_c6f9d16b-cd62-44c3-82fe-0b27a89eb580/monitoring-plugin/0.log" Jul 05 20:11:48.422310 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.422274 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kp72_61302033-be14-4c7c-8834-2248c65f2119/node-exporter/0.log" Jul 05 20:11:48.440301 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.440242 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kp72_61302033-be14-4c7c-8834-2248c65f2119/kube-rbac-proxy/0.log" Jul 05 20:11:48.459411 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.459387 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8kp72_61302033-be14-4c7c-8834-2248c65f2119/init-textfile/0.log" Jul 05 20:11:48.933010 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.932934 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-7t9th_25805d8e-fec2-40e7-ba31-78e90b70e517/prometheus-operator-admission-webhook/0.log" Jul 05 20:11:48.964806 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.964778 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77ff47d74f-dsv2n_740a4233-c5e2-4d54-ae2b-6b93ee0e78b4/telemeter-client/0.log" Jul 05 20:11:48.989107 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:48.989046 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77ff47d74f-dsv2n_740a4233-c5e2-4d54-ae2b-6b93ee0e78b4/reload/0.log" Jul 05 20:11:49.009438 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:49.009404 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77ff47d74f-dsv2n_740a4233-c5e2-4d54-ae2b-6b93ee0e78b4/kube-rbac-proxy/0.log" Jul 05 20:11:51.544109 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.544007 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-675878ffb5-c9gx7_bdf142d2-25f7-40b9-98ae-1d96fd4d9fc3/console/0.log" Jul 05 20:11:51.693353 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.693316 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz"] Jul 05 20:11:51.702034 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.702005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.705751 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.705280 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz"] Jul 05 20:11:51.771370 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.771339 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-proc\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.771563 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.771404 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-sys\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.771563 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.771467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-podres\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.771563 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.771502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-lib-modules\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.771563 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.771534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/2c767843-1986-463e-bbdd-9b4dcca558f9-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873045 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.872950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-sys\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873045 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-podres\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873048 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-lib-modules\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873075 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-sys\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/2c767843-1986-463e-bbdd-9b4dcca558f9-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-proc\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873200 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-podres\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873209 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-lib-modules\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.873287 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.873240 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2c767843-1986-463e-bbdd-9b4dcca558f9-proc\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:51.883294 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:51.883240 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9l7\" (UniqueName: \"kubernetes.io/projected/2c767843-1986-463e-bbdd-9b4dcca558f9-kube-api-access-sj9l7\") pod \"perf-node-gather-daemonset-fqkbz\" (UID: \"2c767843-1986-463e-bbdd-9b4dcca558f9\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:52.018330 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.018097 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:52.216077 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.216012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz"] Jul 05 20:11:52.510163 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.510120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" event={"ID":"2c767843-1986-463e-bbdd-9b4dcca558f9","Type":"ContainerStarted","Data":"a66fe8b73e7e3dde27fea9e2ebe0d62ed3ff29291d76451497a4b40311e4cc0f"} Jul 05 20:11:52.510362 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.510191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" event={"ID":"2c767843-1986-463e-bbdd-9b4dcca558f9","Type":"ContainerStarted","Data":"8690a1fa271a2ecdcf5a5da02fa9799c0df2bb481bc16b730c27ceef59d6ae1d"} Jul 05 20:11:52.510833 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.510813 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:11:52.530476 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:52.530416 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" podStartSLOduration=1.530396687 podStartE2EDuration="1.530396687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 20:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:11:52.525089439 +0000 UTC m=+2512.980872869" watchObservedRunningTime="2026-07-05 20:11:52.530396687 +0000 UTC m=+2512.986180124" Jul 05 20:11:53.340758 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:53.340693 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6wtd_6a6a3896-1443-41de-aa31-60519c14caab/dns/0.log" Jul 05 20:11:53.363992 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:53.363964 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n6wtd_6a6a3896-1443-41de-aa31-60519c14caab/kube-rbac-proxy/0.log" Jul 05 20:11:53.429828 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:53.429804 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jdm75_5a8d4538-2d34-496f-8aec-a84417bda656/dns-node-resolver/0.log" Jul 05 20:11:53.956544 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:53.956498 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d8c5bcbb9-lwplr_e0db61f8-fdb0-43ef-9918-9bf89879c706/registry/0.log" Jul 05 20:11:54.024398 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:54.024365 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jkxjb_c52aea1b-58e4-41ac-ac04-db4a90d9c794/node-ca/0.log" Jul 05 20:11:55.098255 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:55.098223 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-5vhjg_ba0f13ae-ef6b-4aec-9489-9636efa3a0b6/kube-auth-proxy/0.log" Jul 05 20:11:55.282792 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:55.282763 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t26cp_8ca415f0-3600-4e50-8c56-72135202d508/payload-processing/0.log" Jul 05 20:11:55.952577 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:55.952539 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b4dq5_3afe59f0-b120-44a0-b841-4a26e694ce15/serve-healthcheck-canary/0.log" Jul 05 20:11:56.426187 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:56.426156 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ztfh_6b5ddf24-8134-439a-a999-946c99797e24/kube-rbac-proxy/0.log" Jul 05 20:11:56.445098 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:56.445070 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ztfh_6b5ddf24-8134-439a-a999-946c99797e24/exporter/0.log" Jul 05 20:11:56.464707 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:56.464679 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4ztfh_6b5ddf24-8134-439a-a999-946c99797e24/extractor/0.log" Jul 05 20:11:58.644502 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:58.644462 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lfkxp_9963cb83-7685-498e-a64d-5a48e4328664/manager/0.log" Jul 05 20:11:58.811493 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:58.811460 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rzk4c_2b5a1198-388f-42dd-b505-59ffc6c818d5/manager/0.log" Jul 05 20:11:58.996730 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:58.996696 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-mmrr2_8ca41f8c-dca6-46ce-b0cb-0f71d8cd84ed/server/0.log" Jul 05 20:11:59.073653 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:59.073613 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2kbrx_cfb0bcc4-f937-4901-bba7-6d76871b3a73/manager/0.log" Jul 05 20:11:59.162711 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:59.162671 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-648kt_2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6/manager/0.log" Jul 05 20:11:59.178515 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:59.178488 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-648kt_2773d0a4-ce54-4ef6-ab53-aa2fb22b77d6/copy-manifests/0.log" Jul 05 20:11:59.535817 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:11:59.535782 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-fqkbz" Jul 05 20:12:02.065238 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:02.065159 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-5vhjg_ba0f13ae-ef6b-4aec-9489-9636efa3a0b6/kube-auth-proxy/0.log" Jul 05 20:12:02.225729 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:02.225695 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t26cp_8ca415f0-3600-4e50-8c56-72135202d508/payload-processing/0.log" Jul 05 20:12:03.612550 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:03.612476 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-5vhjg_ba0f13ae-ef6b-4aec-9489-9636efa3a0b6/kube-auth-proxy/0.log" Jul 05 20:12:03.767610 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:03.767571 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t26cp_8ca415f0-3600-4e50-8c56-72135202d508/payload-processing/0.log" Jul 05 20:12:06.880093 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.880029 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-756dc_94267a0b-11cf-4f85-8c61-6c7b4bd088ab/kube-multus/0.log" Jul 05 20:12:06.907440 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.907414 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/kube-multus-additional-cni-plugins/0.log" Jul 05 20:12:06.928648 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.928622 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/egress-router-binary-copy/0.log" Jul 05 20:12:06.949646 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.949620 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/cni-plugins/0.log" Jul 05 20:12:06.969127 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.969103 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/bond-cni-plugin/0.log" Jul 05 20:12:06.989236 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:06.989203 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/routeoverride-cni/0.log" Jul 05 20:12:07.012729 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:07.012705 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/whereabouts-cni-bincopy/0.log" Jul 05 20:12:07.031779 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:07.031753 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6wzpw_45c5b67a-a46d-45c7-a3a1-03de771be4fc/whereabouts-cni/0.log" Jul 05 20:12:07.481564 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:07.481530 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tqkck_b0b20ec4-573b-4f2f-9d2a-cf7412f1173e/network-metrics-daemon/0.log" Jul 05 20:12:07.501372 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:07.501344 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tqkck_b0b20ec4-573b-4f2f-9d2a-cf7412f1173e/kube-rbac-proxy/0.log" Jul 05 20:12:08.234462 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.234433 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-controller/0.log" Jul 05 20:12:08.249642 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.249619 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/0.log" Jul 05 20:12:08.249943 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.249928 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovn-acl-logging/1.log" Jul 05 20:12:08.266695 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.266674 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/kube-rbac-proxy-node/0.log" Jul 05 20:12:08.285637 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.285616 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/kube-rbac-proxy-ovn-metrics/0.log" Jul 05 20:12:08.303589 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.303571 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/northd/0.log" Jul 05 20:12:08.324525 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.324496 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/nbdb/0.log" Jul 05 20:12:08.343488 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.343466 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/sbdb/0.log" Jul 05 20:12:08.400113 ip-10-0-143-177 kubenswrapper[2581]: I0705 20:12:08.400083 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5qtff_b26ef5e5-d807-44d7-9d12-905c17073d16/ovnkube-controller/0.log"