Jul 08 17:31:25.132800 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 17:31:25.132815 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 17:31:25.132824 ip-10-0-129-65 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 17:31:25.133145 ip-10-0-129-65 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 17:31:35.204855 ip-10-0-129-65 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 17:31:35.204877 ip-10-0-129-65 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d21604cd9bd64da2951ced99fd84b13e -- Jul 08 17:33:17.084911 ip-10-0-129-65 systemd[1]: Starting Kubernetes Kubelet... Jul 08 17:33:17.523202 ip-10-0-129-65 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:17.523202 ip-10-0-129-65 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 17:33:17.523202 ip-10-0-129-65 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:17.523202 ip-10-0-129-65 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 17:33:17.523202 ip-10-0-129-65 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:17.524269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.524188 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 17:33:17.528986 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.528971 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.528986 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.528986 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.528990 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.528993 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.528996 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529000 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529003 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529005 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529008 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529011 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529014 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529016 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529019 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529025 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529028 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529032 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529036 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529039 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529042 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529045 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529048 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.529047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529050 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529054 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529057 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529066 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529068 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529072 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529074 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529077 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529079 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529082 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529084 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529088 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529092 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529094 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529097 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529099 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529102 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529104 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529107 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529109 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.529527 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529111 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529116 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529120 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529123 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529125 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529129 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529131 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529134 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529136 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529139 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529141 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529143 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529146 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529148 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529152 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529154 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529157 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529159 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529162 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.530019 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529164 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529167 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529169 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529172 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529175 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529177 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529180 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529182 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529184 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529187 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529189 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529192 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529194 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529196 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529199 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529201 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529204 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529206 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529209 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529212 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.530575 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529215 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529217 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529219 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529222 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529224 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529227 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529229 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529233 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529235 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529238 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529240 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529242 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529245 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529247 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529249 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529252 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529254 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529257 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529259 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529962 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.531467 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529970 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529973 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529975 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529978 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529980 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529983 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529987 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529990 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529992 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529995 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.529998 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530000 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530004 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530007 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530011 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530015 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530018 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530021 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530024 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.532072 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530028 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530031 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530033 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530037 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530039 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530042 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530044 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530047 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530049 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530052 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530054 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530057 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530059 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530062 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530065 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530068 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530070 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530073 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530075 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.532543 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530078 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530081 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530083 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530086 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530089 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530091 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530094 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530097 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530099 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530101 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530104 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530106 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530109 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530111 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530113 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530116 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530118 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530120 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530122 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530125 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.533027 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530127 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530131 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530134 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530137 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530140 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530142 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530145 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530147 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530149 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530152 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530154 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530157 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530160 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530163 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530165 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530168 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530170 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530174 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530176 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.533518 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530178 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530181 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530184 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530186 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530188 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530191 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530193 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530196 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530198 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530201 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530203 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530206 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530208 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530211 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530214 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530216 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530218 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530221 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530223 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530226 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.534008 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.530228 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531583 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531612 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531616 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531620 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531622 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531627 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531630 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531633 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531636 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531639 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531641 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531644 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531647 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531649 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531652 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531654 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531657 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531659 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531662 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.534492 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531664 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531667 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531669 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531672 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531674 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531676 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531679 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531681 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531684 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531686 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531688 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531691 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531694 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531700 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531703 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531706 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531708 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531711 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531714 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531716 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.534995 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531718 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531721 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531724 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531726 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531728 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531732 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531735 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531738 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531741 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531744 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531746 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531748 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531751 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531753 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531756 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531758 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531760 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531763 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531765 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531768 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.535495 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531770 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531773 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531775 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531777 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531780 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531783 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531785 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531788 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531790 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531793 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531795 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531798 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531800 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531803 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531806 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531808 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531811 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531813 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531816 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531820 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.535988 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531822 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531825 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531827 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531829 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531832 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531834 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531837 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531839 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531842 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531844 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531847 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531849 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531853 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531857 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531860 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531863 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531866 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531869 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531872 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.531874 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.536510 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531936 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531943 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531949 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531953 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531958 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531961 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531968 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531972 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531975 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531978 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531981 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531984 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531987 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531990 2579 flags.go:64] FLAG: --cgroup-root="" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531992 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531995 2579 flags.go:64] FLAG: --client-ca-file="" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.531999 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532002 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532005 2579 flags.go:64] FLAG: --cluster-domain="" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532008 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532011 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532014 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532017 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532021 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 17:33:17.537042 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532023 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532026 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532029 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532032 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532035 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532038 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532042 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532045 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532049 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532052 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532055 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532057 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532060 2579 flags.go:64] FLAG: --enable-server="true" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532063 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532068 2579 flags.go:64] FLAG: --event-burst="100" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532070 2579 flags.go:64] FLAG: --event-qps="50" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532073 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532076 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532079 2579 flags.go:64] FLAG: --eviction-hard="" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532083 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532085 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532088 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532091 2579 flags.go:64] FLAG: --eviction-soft="" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532094 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532097 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 17:33:17.537636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532101 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532104 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532107 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532110 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532112 2579 flags.go:64] FLAG: --feature-gates="" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532116 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532119 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532121 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532124 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532127 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532130 2579 flags.go:64] FLAG: --help="false" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532135 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532138 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532141 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532144 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532147 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532150 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532153 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532156 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532159 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532162 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532164 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532167 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532170 2579 flags.go:64] FLAG: --kube-reserved="" Jul 08 17:33:17.538284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532172 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532175 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532178 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532180 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532183 2579 flags.go:64] FLAG: --lock-file="" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532186 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532188 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532191 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532196 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532199 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532202 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532205 2579 flags.go:64] FLAG: --logging-format="text" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532207 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532210 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532213 2579 flags.go:64] FLAG: --manifest-url="" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532215 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532219 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532222 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532226 2579 flags.go:64] FLAG: --max-pods="110" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532230 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532233 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532235 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532238 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532241 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532244 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 17:33:17.538917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532247 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532253 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532256 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532259 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532262 2579 flags.go:64] FLAG: --pod-cidr="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532264 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532269 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532272 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532275 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532277 2579 flags.go:64] FLAG: --port="10250" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532280 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532283 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02bcdf5eaf3a948ea" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532286 2579 flags.go:64] FLAG: --qos-reserved="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532289 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532292 2579 flags.go:64] FLAG: --register-node="true" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532294 2579 flags.go:64] FLAG: --register-with-taints="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532298 2579 flags.go:64] FLAG: --registry-burst="10" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532301 2579 flags.go:64] FLAG: --registry-qps="5" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532304 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532306 2579 flags.go:64] FLAG: --reserved-memory="" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532310 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532313 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532315 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532318 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532320 2579 flags.go:64] FLAG: --runonce="false" Jul 08 17:33:17.539524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532323 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532327 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532331 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532334 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532337 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532339 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532342 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532345 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532348 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532350 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532353 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532356 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532359 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532362 2579 flags.go:64] FLAG: --system-cgroups="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532364 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532370 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532373 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532375 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532379 2579 flags.go:64] FLAG: --tls-min-version="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532382 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532385 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532387 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532390 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532393 2579 flags.go:64] FLAG: --v="2" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532397 2579 flags.go:64] FLAG: --version="false" Jul 08 17:33:17.540146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532401 2579 flags.go:64] FLAG: --vmodule="" Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532404 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532407 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532517 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532520 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532523 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532526 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532529 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532533 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532537 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532540 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532543 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532546 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532548 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532551 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532554 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532556 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532559 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532562 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532565 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.540762 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532567 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532570 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532572 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532575 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532578 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532580 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532583 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532585 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532588 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532590 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532610 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532614 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532617 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532620 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532623 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532626 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532629 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532633 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532636 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532638 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.541248 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532642 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532647 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532649 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532652 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532654 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532657 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532660 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532662 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532665 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532667 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532671 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532673 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532675 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532678 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532680 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532683 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532685 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532688 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532690 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532693 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.541778 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532695 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532698 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532700 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532702 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532705 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532708 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532711 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532713 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532715 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532718 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532720 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532723 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532727 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532730 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532733 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532735 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532738 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532740 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532742 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532745 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.542258 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532747 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532749 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532752 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532754 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532757 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532759 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532763 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532766 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532770 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532773 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532776 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532778 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532781 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532784 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532786 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532789 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532791 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532794 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532796 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.542765 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532799 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532801 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.532803 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.532812 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.539974 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.539988 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.540850 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.540857 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540913 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540918 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540922 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540926 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540930 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540933 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.543216 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540936 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540938 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540941 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540945 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540949 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540951 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540954 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540957 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540959 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540961 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540964 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540966 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540969 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540971 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540974 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540976 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540979 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540981 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540984 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.543564 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540987 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540989 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540991 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540994 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.540998 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541000 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541003 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541005 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541008 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541010 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541013 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541016 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541018 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541020 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541023 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541025 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541028 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541031 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541033 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.544031 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541035 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541040 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541043 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541046 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541049 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541051 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541054 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541057 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541060 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541063 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541066 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541069 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541071 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541075 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541077 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541080 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541082 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541085 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541088 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541090 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.544501 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541093 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541095 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541098 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541100 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541103 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541106 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541108 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541110 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541113 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541115 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541118 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541120 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541123 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541125 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541128 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541130 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541133 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541135 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541137 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541140 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.545021 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541143 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541145 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541148 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541150 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541153 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541155 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541159 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541161 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541163 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541166 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541169 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541171 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541173 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541176 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541178 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.545491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.541183 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541288 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541292 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541295 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541298 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541301 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541304 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541306 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541309 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541312 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541315 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541318 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541321 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541323 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541326 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541328 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541331 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541333 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541336 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541338 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:17.545868 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541341 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541343 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541345 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541348 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541350 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541353 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541355 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541358 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541360 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541362 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541365 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541367 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541370 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541372 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541375 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541377 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541380 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541382 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541385 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541387 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:17.546346 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541390 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541392 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541394 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541397 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541399 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541401 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541404 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541406 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541408 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541411 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541413 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541416 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541418 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541421 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541423 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541426 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541428 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541431 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541434 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541436 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:17.546840 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541439 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541441 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541444 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541446 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541448 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541452 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541455 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541458 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541461 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541463 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541466 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541468 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541471 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541473 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541476 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541478 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541480 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541483 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:17.547327 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541487 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541490 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541493 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541495 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541498 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541501 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541503 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541506 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541509 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541511 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541514 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541516 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541519 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541521 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541524 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541526 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541529 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541531 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541534 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541536 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:17.547850 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541539 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:17.541541 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.541546 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.541677 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.544072 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.545743 2579 server.go:1019] "Starting client certificate rotation" Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.545838 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:33:17.548331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.545875 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:33:17.548625 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.548613 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:33:17.549705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.549686 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:33:17.565156 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.564286 2579 log.go:25] "Validated CRI v1 runtime API" Jul 08 17:33:17.567507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.567491 2579 log.go:25] "Validated CRI v1 image API" Jul 08 17:33:17.568331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.568316 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 17:33:17.572915 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.572897 2579 fs.go:135] Filesystem UUIDs: map[3104122e-a279-4e28-b951-627a9c00722b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e96b5d84-e701-4510-b42e-5e8453ef7000:/dev/nvme0n1p4] Jul 08 17:33:17.572976 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.572916 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 17:33:17.582738 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.582628 2579 manager.go:216] Machine: {Timestamp:2026-07-08 17:33:17.582133361 +0000 UTC m=+0.384219871 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3141572 MemoryCapacity:32812183552 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26f130428436778b14bde33e754a12 SystemUUID:ec26f130-4284-3677-8b14-bde33e754a12 BootID:d21604cd-9bd6-4da2-951c-ed99fd84b13e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406093824 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:1d:a5:16:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:1d:a5:16:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:0c:d9:2b:32:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812183552 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 17:33:17.582738 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.582726 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 17:33:17.582853 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.582795 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 17:33:17.583900 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.583884 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:33:17.586561 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.586529 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 17:33:17.586711 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.586563 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 17:33:17.587408 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.587399 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 17:33:17.587442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.587411 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 17:33:17.587442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.587418 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:33:17.588335 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.588315 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:33:17.590137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.590127 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 17:33:17.594373 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594362 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:33:17.594490 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594481 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 17:33:17.594519 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594511 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 08 17:33:17.594555 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594533 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 17:33:17.594584 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594563 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 17:33:17.594584 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594576 2579 kubelet.go:398] "Adding apiserver pod source" Jul 08 17:33:17.594658 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.594588 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 17:33:17.595015 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.595004 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:33:17.595091 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.595022 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:33:17.597506 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.597490 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 17:33:17.599771 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.599759 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 17:33:17.599812 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.599789 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 17:33:17.601304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601291 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 17:33:17.601304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601307 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601313 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601319 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601329 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601334 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601342 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601347 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601353 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601359 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601372 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 17:33:17.601405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.601377 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 17:33:17.603088 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.603075 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 17:33:17.603088 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.603089 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 17:33:17.606413 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.606399 2579 server.go:1268] "Started kubelet" Jul 08 17:33:17.606624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.606551 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 17:33:17.606699 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.606653 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 17:33:17.607068 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.607027 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 17:33:17.607068 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.607035 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 17:33:17.607192 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.607157 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 17:33:17.607270 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.607243 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 17:33:17.607469 ip-10-0-129-65 systemd[1]: Started Kubernetes Kubelet. Jul 08 17:33:17.608808 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.608793 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 17:33:17.610960 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.610939 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 08 17:33:17.614923 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.614896 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 17:33:17.616256 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.615161 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-65.ec2.internal.18c060e6faded2d7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.ec2.internal,UID:ip-10-0-129-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.ec2.internal,},FirstTimestamp:2026-07-08 17:33:17.606367959 +0000 UTC m=+0.408454470,LastTimestamp:2026-07-08 17:33:17.606367959 +0000 UTC m=+0.408454470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.ec2.internal,}" Jul 08 17:33:17.617375 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.617356 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 17:33:17.617375 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.617365 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 17:33:17.617515 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.617444 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:17.618191 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618173 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 17:33:17.618266 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618194 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 17:33:17.618346 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618195 2579 factory.go:55] Registering systemd factory Jul 08 17:33:17.618404 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618349 2579 factory.go:223] Registration of the systemd container factory successfully Jul 08 17:33:17.618404 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618386 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 17:33:17.618496 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618475 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 08 17:33:17.618496 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618486 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 08 17:33:17.618583 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618558 2579 factory.go:153] Registering CRI-O factory Jul 08 17:33:17.618583 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618572 2579 factory.go:223] Registration of the crio container factory successfully Jul 08 17:33:17.618705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618650 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 17:33:17.618705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618667 2579 factory.go:103] Registering Raw factory Jul 08 17:33:17.618705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.618680 2579 manager.go:1196] Started watching for new ooms in manager Jul 08 17:33:17.619112 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.619101 2579 manager.go:349] Starting recovery of all containers Jul 08 17:33:17.619967 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.619894 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 17:33:17.620186 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.619979 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 17:33:17.628144 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.627982 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nh6j4" Jul 08 17:33:17.629008 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.628991 2579 manager.go:354] Recovery completed Jul 08 17:33:17.630827 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.630801 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 08 17:33:17.634210 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.634197 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.636047 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636032 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.636110 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636065 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.636110 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636075 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.636545 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636532 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 17:33:17.636545 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636542 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 17:33:17.636545 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636545 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.636670 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.636557 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:33:17.637194 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637181 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.637247 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637203 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.637247 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637212 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.637486 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637473 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nh6j4" Jul 08 17:33:17.637675 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637664 2579 policy_none.go:49] "None policy: Start" Jul 08 17:33:17.637710 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637679 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 17:33:17.637710 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.637689 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 17:33:17.638358 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.638293 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-65.ec2.internal.18c060e6fca3b66c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-65.ec2.internal,UID:ip-10-0-129-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-65.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-65.ec2.internal,},FirstTimestamp:2026-07-08 17:33:17.636048492 +0000 UTC m=+0.438135006,LastTimestamp:2026-07-08 17:33:17.636048492 +0000 UTC m=+0.438135006,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-65.ec2.internal,}" Jul 08 17:33:17.639045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.639025 2579 policy_none.go:47] "Start" Jul 08 17:33:17.678176 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678162 2579 manager.go:337] "Starting Device Plugin manager" Jul 08 17:33:17.678283 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.678195 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 17:33:17.678283 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678205 2579 server.go:87] "Starting device plugin registration server" Jul 08 17:33:17.678445 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678433 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 17:33:17.678925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678449 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 17:33:17.678925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678630 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 17:33:17.678925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678699 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 17:33:17.678925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.678706 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 17:33:17.681247 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.681225 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 17:33:17.694520 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.681284 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:17.779084 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.779007 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.780164 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.780143 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.780273 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.780174 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.780273 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.780185 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.780273 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.780210 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.783225 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.783203 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 17:33:17.784560 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.784542 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 17:33:17.784636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.784589 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 17:33:17.784636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.784632 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 17:33:17.784710 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.784673 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 17:33:17.787075 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.787053 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:17.792179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.792159 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.792252 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.792180 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-65.ec2.internal\": node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:17.831696 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.831675 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:17.884992 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.884969 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal"] Jul 08 17:33:17.885066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.885028 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.885851 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.885835 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.885912 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.885871 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.885912 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.885887 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.887164 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887152 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.887287 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.887321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887305 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.887817 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887802 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.887895 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887824 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.887895 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887838 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.887996 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887901 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.887996 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887927 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.887996 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.887942 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.888969 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.888954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.889047 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.888983 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:17.889699 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.889682 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:17.889758 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.889717 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:17.889758 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.889730 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:17.911296 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.911275 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-65.ec2.internal\" not found" node="ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.915528 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.915511 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-65.ec2.internal\" not found" node="ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.920630 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.920611 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.920698 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.920642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.920698 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:17.920659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94467296c0cab4e5954502ea1cc5498e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"94467296c0cab4e5954502ea1cc5498e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:17.931920 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:17.931905 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.021635 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.021635 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.021784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94467296c0cab4e5954502ea1cc5498e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"94467296c0cab4e5954502ea1cc5498e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.021784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021699 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.021784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aca91109daef0c5ad11720d851b32cd8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal\" (UID: \"aca91109daef0c5ad11720d851b32cd8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.021784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.021704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94467296c0cab4e5954502ea1cc5498e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-65.ec2.internal\" (UID: \"94467296c0cab4e5954502ea1cc5498e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.032739 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.032692 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.133477 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.133453 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.214678 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.214644 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.219423 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.219407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.234115 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.234099 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.334643 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.334565 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.435136 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.435110 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.535610 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.535567 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-65.ec2.internal\" not found" Jul 08 17:33:18.544732 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.544713 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 17:33:18.544856 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.544839 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 17:33:18.583354 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.583337 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:18.595447 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.595400 2579 apiserver.go:52] "Watching apiserver" Jul 08 17:33:18.604798 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.604778 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 17:33:18.605096 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.605076 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qk9v6","openshift-multus/multus-pk4h7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd","openshift-cluster-node-tuning-operator/tuned-8jn7h","openshift-multus/network-metrics-daemon-nmqqm","openshift-network-diagnostics/network-check-target-75zck","openshift-network-operator/iptables-alerter-kbr5b","openshift-ovn-kubernetes/ovnkube-node-nlrfb","kube-system/global-pull-secret-syncer-7x7wz","kube-system/konnectivity-agent-x66br","openshift-dns/node-resolver-kthzc","openshift-image-registry/node-ca-2kz8j"] Jul 08 17:33:18.606367 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.606352 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.607353 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.607334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.609534 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.609512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.609734 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.609707 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.609875 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.609782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.609875 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.609854 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 17:33:18.610005 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.609989 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 17:33:18.610583 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.610562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 17:33:18.610733 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.610655 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v2gbg\"" Jul 08 17:33:18.610733 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.610664 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 17:33:18.610848 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.610754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.610848 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.610841 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7m8tw\"" Jul 08 17:33:18.611837 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.611823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.611949 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.611932 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:18.612553 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.612537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.612738 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.612723 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkskc\"" Jul 08 17:33:18.612975 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.612954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:18.613062 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613005 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 17:33:18.613062 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.613062 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.613033 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:18.613191 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613122 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.613310 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613286 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.613381 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613313 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nmqqq\"" Jul 08 17:33:18.614011 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.613996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.615108 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.615093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.615932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.615878 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 17:33:18.616180 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.616162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.616274 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.616204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jwrcm\"" Jul 08 17:33:18.616274 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.616213 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.616274 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.616170 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.616445 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.616342 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:18.617120 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617101 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.617328 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617311 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.617392 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617317 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 17:33:18.617392 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617351 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 17:33:18.617489 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.617489 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617474 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 17:33:18.617645 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617567 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.617791 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617567 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 17:33:18.617864 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.617846 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdql8\"" Jul 08 17:33:18.618072 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.618052 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 17:33:18.619080 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.619064 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.620040 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.620024 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 17:33:18.620143 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.620041 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 17:33:18.620314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.620299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8rs65\"" Jul 08 17:33:18.620391 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.620375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.621579 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.621564 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2svwm\"" Jul 08 17:33:18.621675 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.621628 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.621724 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.621631 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.622407 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.622392 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 17:33:18.622509 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.622488 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 17:33:18.622579 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.622510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 17:33:18.622746 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.622734 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c4cn\"" Jul 08 17:33:18.624506 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-system-cni-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.624590 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-kubelet\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.624590 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-var-lib-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.624590 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-socket-dir-parent\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-tmp\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72m7f\" (UniqueName: \"kubernetes.io/projected/5997a67d-d6be-44d7-be12-0c42de6ded38-kube-api-access-72m7f\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-etc-selinux\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-os-release\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-etc-kubernetes\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.624769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn8pm\" (UniqueName: \"kubernetes.io/projected/fdf27821-1ca3-488b-9ca9-72778533b2d0-kube-api-access-rn8pm\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5997a67d-d6be-44d7-be12-0c42de6ded38-tmp-dir\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1702b59-fac0-4bdb-b651-5d08a675ee86-iptables-alerter-script\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1702b59-fac0-4bdb-b651-5d08a675ee86-host-slash\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.624983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-etc-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-node-log\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625045 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-cnibin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-cni-binary-copy\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-os-release\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-script-lib\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-conf-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-tuned\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-log-socket\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-k8s-cni-cncf-io\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-kubernetes\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-host\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-bin\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-config\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-multus\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625383 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmp8\" (UniqueName: \"kubernetes.io/projected/1ee437b7-95ac-4fbb-9895-f1acb4341051-kube-api-access-skmp8\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5997a67d-d6be-44d7-be12-0c42de6ded38-hosts-file\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-systemd\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-netns\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-sys\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-var-lib-kubelet\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-systemd-units\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-system-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-daemon-config\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-modprobe-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-run\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gjn\" (UniqueName: \"kubernetes.io/projected/2170557d-59b8-4c8a-8575-d9ca35586250-kube-api-access-z7gjn\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-device-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.625910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-sys-fs\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-ovn\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-bin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhkl\" (UniqueName: \"kubernetes.io/projected/fa2450d5-725a-40c0-9c41-0ea21e082328-kube-api-access-lwhkl\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r72p\" (UniqueName: \"kubernetes.io/projected/f1702b59-fac0-4bdb-b651-5d08a675ee86-kube-api-access-2r72p\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-cnibin\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-socket-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-registration-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-conf\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.625981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-kubelet-config\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-konnectivity-ca\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-systemd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-kubelet\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysconfig\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-agent-certs\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.626385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-multus-certs\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-dbus\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-env-overrides\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vzb\" (UniqueName: \"kubernetes.io/projected/d1bc455f-6a8a-4c83-aef5-022768f814f0-kube-api-access-n2vzb\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-netd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovn-node-metrics-cert\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-slash\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-netns\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-lib-modules\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdhh\" (UniqueName: \"kubernetes.io/projected/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-kube-api-access-xbdhh\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.626858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-hostroot\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.627299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.626990 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal"] Jul 08 17:33:18.627299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.627021 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:33:18.627299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.627069 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" Jul 08 17:33:18.633493 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.633476 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:33:18.639266 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.639210 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 17:28:18 +0000 UTC" deadline="2028-04-08 13:07:26.514407006 +0000 UTC" Jul 08 17:33:18.639266 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.639260 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15355h34m7.875151184s" Jul 08 17:33:18.645138 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.645123 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:33:18.645290 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.645276 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal"] Jul 08 17:33:18.654191 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.654177 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9bnhj" Jul 08 17:33:18.661049 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.661033 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9bnhj" Jul 08 17:33:18.719450 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.719429 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 17:33:18.726617 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-daemon-config\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.726730 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-modprobe-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.726730 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-run\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.726730 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gjn\" (UniqueName: \"kubernetes.io/projected/2170557d-59b8-4c8a-8575-d9ca35586250-kube-api-access-z7gjn\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-device-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-sys-fs\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-ovn\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726756 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-run\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-device-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.726840 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-sys-fs\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-modprobe-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726870 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-ovn\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-bin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.726885 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-bin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.726910 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:19.226890791 +0000 UTC m=+2.028977293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhkl\" (UniqueName: \"kubernetes.io/projected/fa2450d5-725a-40c0-9c41-0ea21e082328-kube-api-access-lwhkl\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r72p\" (UniqueName: \"kubernetes.io/projected/f1702b59-fac0-4bdb-b651-5d08a675ee86-kube-api-access-2r72p\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-cnibin\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.726993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-socket-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-registration-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727033 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-cnibin\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-daemon-config\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-conf\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-registration-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-kubelet-config\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-socket-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727135 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-konnectivity-ca\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727153 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-kubelet-config\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.727446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-systemd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-d\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727183 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-systemd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-kubelet\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysconfig\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysctl-conf\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-agent-certs\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727240 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-kubelet\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-sysconfig\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-multus-certs\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727352 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-dbus\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-env-overrides\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727408 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-multus-certs\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vzb\" (UniqueName: \"kubernetes.io/projected/d1bc455f-6a8a-4c83-aef5-022768f814f0-kube-api-access-n2vzb\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-netd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovn-node-metrics-cert\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-slash\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727498 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-dbus\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727498 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-slash\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-netd\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727659 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-konnectivity-ca\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-netns\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-netns\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727807 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-binary-copy\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-lib-modules\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-env-overrides\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.728769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-host\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdhh\" (UniqueName: \"kubernetes.io/projected/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-kube-api-access-xbdhh\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-hostroot\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-lib-modules\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-system-cni-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.727988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-kubelet\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728004 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-hostroot\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-var-lib-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-system-cni-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-socket-dir-parent\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-tmp\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728084 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-socket-dir-parent\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72m7f\" (UniqueName: \"kubernetes.io/projected/5997a67d-d6be-44d7-be12-0c42de6ded38-kube-api-access-72m7f\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-kubelet\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-var-lib-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-etc-selinux\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.729705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-etc-selinux\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-os-release\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-os-release\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-etc-kubernetes\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn8pm\" (UniqueName: \"kubernetes.io/projected/fdf27821-1ca3-488b-9ca9-72778533b2d0-kube-api-access-rn8pm\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728326 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-run-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5997a67d-d6be-44d7-be12-0c42de6ded38-tmp-dir\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1702b59-fac0-4bdb-b651-5d08a675ee86-iptables-alerter-script\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728529 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1702b59-fac0-4bdb-b651-5d08a675ee86-host-slash\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728551 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ee437b7-95ac-4fbb-9895-f1acb4341051-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.728625 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728656 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-etc-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728708 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-etc-kubernetes\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.730507 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.728720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:19.228704173 +0000 UTC m=+2.030790686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-etc-openvswitch\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-node-log\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-cnibin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-cni-binary-copy\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5rr\" (UniqueName: \"kubernetes.io/projected/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-kube-api-access-7q5rr\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-os-release\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.728949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-script-lib\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729004 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2170557d-59b8-4c8a-8575-d9ca35586250-os-release\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1702b59-fac0-4bdb-b651-5d08a675ee86-host-slash\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-conf-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-node-log\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-tuned\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-cnibin\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-log-socket\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1702b59-fac0-4bdb-b651-5d08a675ee86-iptables-alerter-script\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729149 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-multus-conf-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-log-socket\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-k8s-cni-cncf-io\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-kubernetes\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-k8s-cni-cncf-io\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-host\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729347 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5997a67d-d6be-44d7-be12-0c42de6ded38-tmp-dir\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-bin\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-config\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729434 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-host\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729470 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-kubernetes\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-multus\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmp8\" (UniqueName: \"kubernetes.io/projected/1ee437b7-95ac-4fbb-9895-f1acb4341051-kube-api-access-skmp8\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729583 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2450d5-725a-40c0-9c41-0ea21e082328-cni-binary-copy\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5997a67d-d6be-44d7-be12-0c42de6ded38-hosts-file\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-script-lib\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.731737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-systemd\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-host-cni-bin\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.729962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-var-lib-cni-multus\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730100 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovnkube-config\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5997a67d-d6be-44d7-be12-0c42de6ded38-hosts-file\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2170557d-59b8-4c8a-8575-d9ca35586250-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-systemd\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-netns\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-sys\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-host-run-netns\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-var-lib-kubelet\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-serviceca\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730352 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-var-lib-kubelet\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-systemd-units\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdf27821-1ca3-488b-9ca9-72778533b2d0-sys\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-system-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-systemd-units\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-tmp\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.732231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2450d5-725a-40c0-9c41-0ea21e082328-system-cni-dir\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.732794 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-ovn-node-metrics-cert\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.732794 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.730900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a76373e-ad42-4a73-b723-1cb32ed2f2b7-agent-certs\") pod \"konnectivity-agent-x66br\" (UID: \"8a76373e-ad42-4a73-b723-1cb32ed2f2b7\") " pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.732794 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.731488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fdf27821-1ca3-488b-9ca9-72778533b2d0-etc-tuned\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.741135 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.741116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhkl\" (UniqueName: \"kubernetes.io/projected/fa2450d5-725a-40c0-9c41-0ea21e082328-kube-api-access-lwhkl\") pod \"multus-pk4h7\" (UID: \"fa2450d5-725a-40c0-9c41-0ea21e082328\") " pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.741835 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.741564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gjn\" (UniqueName: \"kubernetes.io/projected/2170557d-59b8-4c8a-8575-d9ca35586250-kube-api-access-z7gjn\") pod \"multus-additional-cni-plugins-qk9v6\" (UID: \"2170557d-59b8-4c8a-8575-d9ca35586250\") " pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.741835 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.741791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r72p\" (UniqueName: \"kubernetes.io/projected/f1702b59-fac0-4bdb-b651-5d08a675ee86-kube-api-access-2r72p\") pod \"iptables-alerter-kbr5b\" (UID: \"f1702b59-fac0-4bdb-b651-5d08a675ee86\") " pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:18.742205 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.742189 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:18.742285 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.742212 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:18.742285 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.742224 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:18.742393 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:18.742289 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:19.242271666 +0000 UTC m=+2.044358175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:18.742515 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.742495 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vzb\" (UniqueName: \"kubernetes.io/projected/d1bc455f-6a8a-4c83-aef5-022768f814f0-kube-api-access-n2vzb\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:18.744846 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.744822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72m7f\" (UniqueName: \"kubernetes.io/projected/5997a67d-d6be-44d7-be12-0c42de6ded38-kube-api-access-72m7f\") pod \"node-resolver-kthzc\" (UID: \"5997a67d-d6be-44d7-be12-0c42de6ded38\") " pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.744981 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.744859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdhh\" (UniqueName: \"kubernetes.io/projected/1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6-kube-api-access-xbdhh\") pod \"ovnkube-node-nlrfb\" (UID: \"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:18.745072 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.745054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmp8\" (UniqueName: \"kubernetes.io/projected/1ee437b7-95ac-4fbb-9895-f1acb4341051-kube-api-access-skmp8\") pod \"aws-ebs-csi-driver-node-7ztsd\" (UID: \"1ee437b7-95ac-4fbb-9895-f1acb4341051\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.745262 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.745246 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn8pm\" (UniqueName: \"kubernetes.io/projected/fdf27821-1ca3-488b-9ca9-72778533b2d0-kube-api-access-rn8pm\") pod \"tuned-8jn7h\" (UID: \"fdf27821-1ca3-488b-9ca9-72778533b2d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:18.753069 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.753054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kthzc" Jul 08 17:33:18.831218 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.831188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-host\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.831375 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.831241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5rr\" (UniqueName: \"kubernetes.io/projected/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-kube-api-access-7q5rr\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.831375 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.831271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-serviceca\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.831375 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.831298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-host\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.831780 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.831759 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-serviceca\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.839363 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.839343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5rr\" (UniqueName: \"kubernetes.io/projected/f80c2ab9-7f70-4fb4-a04d-6c865232a4a7-kube-api-access-7q5rr\") pod \"node-ca-2kz8j\" (UID: \"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7\") " pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:18.844315 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.844298 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:18.872836 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.872810 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94467296c0cab4e5954502ea1cc5498e.slice/crio-73cf7c86097bcca6ae32f5833aa45e19b09b62208825173c9d2dff4e5c93f475 WatchSource:0}: Error finding container 73cf7c86097bcca6ae32f5833aa45e19b09b62208825173c9d2dff4e5c93f475: Status 404 returned error can't find the container with id 73cf7c86097bcca6ae32f5833aa45e19b09b62208825173c9d2dff4e5c93f475 Jul 08 17:33:18.873906 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.873886 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca91109daef0c5ad11720d851b32cd8.slice/crio-378aceadbd2c78d930f5aca1c3e322d1a02031a2d82f5c24775b44aa97c06f46 WatchSource:0}: Error finding container 378aceadbd2c78d930f5aca1c3e322d1a02031a2d82f5c24775b44aa97c06f46: Status 404 returned error can't find the container with id 378aceadbd2c78d930f5aca1c3e322d1a02031a2d82f5c24775b44aa97c06f46 Jul 08 17:33:18.877092 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.877067 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:33:18.926242 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.926219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" Jul 08 17:33:18.935366 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.935345 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2170557d_59b8_4c8a_8575_d9ca35586250.slice/crio-fc428a316fd6a8e6fa7101ae9ccb93304e2d7b5a82fd845e6afc63901b5f1553 WatchSource:0}: Error finding container fc428a316fd6a8e6fa7101ae9ccb93304e2d7b5a82fd845e6afc63901b5f1553: Status 404 returned error can't find the container with id fc428a316fd6a8e6fa7101ae9ccb93304e2d7b5a82fd845e6afc63901b5f1553 Jul 08 17:33:18.944945 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.944928 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:18.945940 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.945924 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:18.952900 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.952882 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a76373e_ad42_4a73_b723_1cb32ed2f2b7.slice/crio-f107829737ab94d463c2616f6d028abb47165b1b22a6d07faa72f3ee651b0fe0 WatchSource:0}: Error finding container f107829737ab94d463c2616f6d028abb47165b1b22a6d07faa72f3ee651b0fe0: Status 404 returned error can't find the container with id f107829737ab94d463c2616f6d028abb47165b1b22a6d07faa72f3ee651b0fe0 Jul 08 17:33:18.961956 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.961940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pk4h7" Jul 08 17:33:18.969318 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.969299 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2450d5_725a_40c0_9c41_0ea21e082328.slice/crio-d881193e525d1ffd64366e9e7666917d44ee4f5452043b436d52e3e5dc414ef7 WatchSource:0}: Error finding container d881193e525d1ffd64366e9e7666917d44ee4f5452043b436d52e3e5dc414ef7: Status 404 returned error can't find the container with id d881193e525d1ffd64366e9e7666917d44ee4f5452043b436d52e3e5dc414ef7 Jul 08 17:33:18.983314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.983296 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" Jul 08 17:33:18.991191 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:18.991169 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee437b7_95ac_4fbb_9895_f1acb4341051.slice/crio-7cce6a48503696cd355f4cd617cb31c21e9e86eba514e467b96f3895b6f28a7e WatchSource:0}: Error finding container 7cce6a48503696cd355f4cd617cb31c21e9e86eba514e467b96f3895b6f28a7e: Status 404 returned error can't find the container with id 7cce6a48503696cd355f4cd617cb31c21e9e86eba514e467b96f3895b6f28a7e Jul 08 17:33:18.994900 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:18.994886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" Jul 08 17:33:19.001011 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.000996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kbr5b" Jul 08 17:33:19.002392 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:19.002372 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf27821_1ca3_488b_9ca9_72778533b2d0.slice/crio-ce6b7464ec29caa3e2167bd8033c8f98136007251fce1dc1161d8de5ad1c6d2b WatchSource:0}: Error finding container ce6b7464ec29caa3e2167bd8033c8f98136007251fce1dc1161d8de5ad1c6d2b: Status 404 returned error can't find the container with id ce6b7464ec29caa3e2167bd8033c8f98136007251fce1dc1161d8de5ad1c6d2b Jul 08 17:33:19.010110 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:19.010092 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1702b59_fac0_4bdb_b651_5d08a675ee86.slice/crio-6dacd126b97df1256cc13e22452d7146ae06e1dc64d4ecfbc760bf865e94b1d0 WatchSource:0}: Error finding container 6dacd126b97df1256cc13e22452d7146ae06e1dc64d4ecfbc760bf865e94b1d0: Status 404 returned error can't find the container with id 6dacd126b97df1256cc13e22452d7146ae06e1dc64d4ecfbc760bf865e94b1d0 Jul 08 17:33:19.020408 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.020390 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:19.028785 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:19.028766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb051d6_ad34_49e8_bf5c_cb6b3fc2d1d6.slice/crio-db9f48d91363d07032244a9f539cf44d3028df8477d15c499452417c0e817ee2 WatchSource:0}: Error finding container db9f48d91363d07032244a9f539cf44d3028df8477d15c499452417c0e817ee2: Status 404 returned error can't find the container with id db9f48d91363d07032244a9f539cf44d3028df8477d15c499452417c0e817ee2 Jul 08 17:33:19.055478 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.055460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2kz8j" Jul 08 17:33:19.063873 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:19.063851 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80c2ab9_7f70_4fb4_a04d_6c865232a4a7.slice/crio-e3c934a0da4f69076cb82613a8ec2b7629c74711aa7e32d92194a123c38468ff WatchSource:0}: Error finding container e3c934a0da4f69076cb82613a8ec2b7629c74711aa7e32d92194a123c38468ff: Status 404 returned error can't find the container with id e3c934a0da4f69076cb82613a8ec2b7629c74711aa7e32d92194a123c38468ff Jul 08 17:33:19.210686 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:19.210649 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5997a67d_d6be_44d7_be12_0c42de6ded38.slice/crio-9e0ce27f537088789739880bddf480da248544fb350b24d7cdb1e0636107aeb2 WatchSource:0}: Error finding container 9e0ce27f537088789739880bddf480da248544fb350b24d7cdb1e0636107aeb2: Status 404 returned error can't find the container with id 9e0ce27f537088789739880bddf480da248544fb350b24d7cdb1e0636107aeb2 Jul 08 17:33:19.234811 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.234781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:19.234916 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.234841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:19.234982 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.234964 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:19.234982 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.234978 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:19.235067 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.235033 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:20.235015707 +0000 UTC m=+3.037102205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:19.235067 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.235048 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:20.23504181 +0000 UTC m=+3.037128310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:19.335170 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.335137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:19.335361 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.335341 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:19.335436 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.335369 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:19.335436 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.335382 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:19.335553 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.335453 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:20.335431747 +0000 UTC m=+3.137518249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:19.663368 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.663280 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:28:19 +0000 UTC" deadline="2028-01-23 16:39:46.930175734 +0000 UTC" Jul 08 17:33:19.663368 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.663321 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13535h6m27.266859332s" Jul 08 17:33:19.785390 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.785361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:19.785566 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:19.785464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:19.797892 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.797766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"aca91109daef0c5ad11720d851b32cd8","Type":"ContainerStarted","Data":"378aceadbd2c78d930f5aca1c3e322d1a02031a2d82f5c24775b44aa97c06f46"} Jul 08 17:33:19.813638 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.813596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"94467296c0cab4e5954502ea1cc5498e","Type":"ContainerStarted","Data":"73cf7c86097bcca6ae32f5833aa45e19b09b62208825173c9d2dff4e5c93f475"} Jul 08 17:33:19.818348 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.818313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kthzc" event={"ID":"5997a67d-d6be-44d7-be12-0c42de6ded38","Type":"ContainerStarted","Data":"9e0ce27f537088789739880bddf480da248544fb350b24d7cdb1e0636107aeb2"} Jul 08 17:33:19.826345 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.826310 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kz8j" event={"ID":"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7","Type":"ContainerStarted","Data":"e3c934a0da4f69076cb82613a8ec2b7629c74711aa7e32d92194a123c38468ff"} Jul 08 17:33:19.833943 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.833910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" event={"ID":"fdf27821-1ca3-488b-9ca9-72778533b2d0","Type":"ContainerStarted","Data":"ce6b7464ec29caa3e2167bd8033c8f98136007251fce1dc1161d8de5ad1c6d2b"} Jul 08 17:33:19.852232 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.852191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" event={"ID":"1ee437b7-95ac-4fbb-9895-f1acb4341051","Type":"ContainerStarted","Data":"7cce6a48503696cd355f4cd617cb31c21e9e86eba514e467b96f3895b6f28a7e"} Jul 08 17:33:19.858842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.858811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pk4h7" event={"ID":"fa2450d5-725a-40c0-9c41-0ea21e082328","Type":"ContainerStarted","Data":"d881193e525d1ffd64366e9e7666917d44ee4f5452043b436d52e3e5dc414ef7"} Jul 08 17:33:19.860414 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.860385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x66br" event={"ID":"8a76373e-ad42-4a73-b723-1cb32ed2f2b7","Type":"ContainerStarted","Data":"f107829737ab94d463c2616f6d028abb47165b1b22a6d07faa72f3ee651b0fe0"} Jul 08 17:33:19.867755 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.867727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"db9f48d91363d07032244a9f539cf44d3028df8477d15c499452417c0e817ee2"} Jul 08 17:33:19.879310 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.879275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbr5b" event={"ID":"f1702b59-fac0-4bdb-b651-5d08a675ee86","Type":"ContainerStarted","Data":"6dacd126b97df1256cc13e22452d7146ae06e1dc64d4ecfbc760bf865e94b1d0"} Jul 08 17:33:19.893043 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.893010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerStarted","Data":"fc428a316fd6a8e6fa7101ae9ccb93304e2d7b5a82fd845e6afc63901b5f1553"} Jul 08 17:33:19.951989 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:19.951911 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:20.243187 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.243094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:20.243366 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.243190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:20.243430 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.243366 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:20.243488 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.243435 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:22.243414399 +0000 UTC m=+5.045500913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:20.243900 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.243877 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:20.243991 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.243937 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:22.243921946 +0000 UTC m=+5.046008448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:20.344521 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.344480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:20.344779 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.344756 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:20.344861 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.344790 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:20.344861 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.344806 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:20.344986 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.344868 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:22.344847366 +0000 UTC m=+5.146933866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:20.664127 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.664004 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:28:19 +0000 UTC" deadline="2028-01-14 02:43:45.743125186 +0000 UTC" Jul 08 17:33:20.664127 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.664073 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13305h10m25.079056862s" Jul 08 17:33:20.785274 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.785241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:20.785453 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.785380 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:20.785916 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:20.785888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:20.786024 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:20.786005 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:21.788516 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:21.787287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:21.788516 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:21.787689 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:22.264704 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:22.264615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:22.264704 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:22.264701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:22.264942 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.264837 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:22.264942 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.264845 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:22.264942 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.264899 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:26.264879284 +0000 UTC m=+9.066965799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:22.264942 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.264919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:26.264907937 +0000 UTC m=+9.066994436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:22.365107 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:22.365064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:22.365297 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.365271 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:22.365375 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.365307 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:22.365375 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.365322 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:22.365495 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.365393 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:26.365371725 +0000 UTC m=+9.167458235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:22.785304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:22.785268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:22.785497 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.785422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:22.785835 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:22.785814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:22.785958 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:22.785937 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:23.786416 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:23.785861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:23.786416 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:23.786020 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:24.785776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:24.785285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:24.785776 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:24.785409 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:24.785776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:24.785455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:24.785776 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:24.785598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:25.785792 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:25.785752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:25.786229 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:25.785913 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:26.302086 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:26.302048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:26.302261 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:26.302131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:26.302261 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.302243 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:26.302261 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.302252 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:26.302436 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.302333 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:34.302311872 +0000 UTC m=+17.104398373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:26.302436 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.302356 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:34.302344588 +0000 UTC m=+17.104431087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:26.403246 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:26.403205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:26.403446 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.403421 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:26.403513 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.403454 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:26.403513 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.403469 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:26.403640 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.403542 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:34.403523421 +0000 UTC m=+17.205609945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:26.785890 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:26.785835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:26.786355 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.785978 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:26.786355 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:26.786295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:26.786448 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:26.786393 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:27.788062 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:27.788024 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:27.788512 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:27.788145 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:28.784918 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:28.784881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:28.785133 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:28.785039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:28.789765 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:28.789727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:28.790187 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:28.789905 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:29.787211 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:29.786737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:29.787211 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:29.786875 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:30.785841 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:30.785746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:30.786211 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:30.785893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:30.786338 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:30.786253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:30.786398 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:30.786349 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:31.785026 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:31.785002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:31.785155 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:31.785106 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:32.784901 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:32.784867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:32.785299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:32.784941 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:32.785299 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:32.784999 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:32.785299 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:32.785097 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:33.784965 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:33.784926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:33.785359 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:33.785070 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:34.368626 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:34.368574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:34.368822 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:34.368655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:34.368822 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.368769 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:34.368933 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.368830 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:50.368810858 +0000 UTC m=+33.170897356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:34.369303 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.369218 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:34.369303 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.369277 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:33:50.369263689 +0000 UTC m=+33.171350186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:34.469879 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:34.469847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:34.470048 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.470028 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:34.470107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.470050 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:34.470107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.470060 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:34.470209 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.470110 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:50.470094557 +0000 UTC m=+33.272181055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:34.785099 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:34.785022 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:34.785099 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:34.785050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:34.785674 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.785158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:34.785674 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:34.785306 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:35.785207 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:35.785172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:35.785613 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:35.785285 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:36.785830 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:36.785798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:36.786270 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:36.785800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:36.786270 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:36.785918 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:36.786270 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:36.786037 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:37.786336 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:37.786298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:37.786727 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:37.786420 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:38.785805 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:38.785767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:38.785996 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:38.785818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:38.785996 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:38.785888 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:38.786107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:38.786036 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:39.785804 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:39.785770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:39.786274 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:39.785904 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:40.785570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.785543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:40.785694 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:40.785673 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:40.785764 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.785757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:40.785852 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:40.785835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:40.955565 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.953444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" event={"ID":"fdf27821-1ca3-488b-9ca9-72778533b2d0","Type":"ContainerStarted","Data":"8c18ee4308f3c772ce11dd5e09616af3dbe58bbcb602733b6c4edf2a22ebc381"} Jul 08 17:33:40.956991 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.956962 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pk4h7" event={"ID":"fa2450d5-725a-40c0-9c41-0ea21e082328","Type":"ContainerStarted","Data":"ed9d44699ff0fdbaeae13cae6b0c59734ef8bc85e022c5c2fcbfeb050ed0d1b8"} Jul 08 17:33:40.960456 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.960424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"5210cc358acee357c41ab68966262149e20a991a104dcc4ddb843b289a337ec6"} Jul 08 17:33:40.968568 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.968197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" event={"ID":"94467296c0cab4e5954502ea1cc5498e","Type":"ContainerStarted","Data":"47b4708fc623baa70e9d002d5e95832cc8cbbbd508020655092669f4be685120"} Jul 08 17:33:40.975747 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:40.975515 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8jn7h" podStartSLOduration=1.194610018 podStartE2EDuration="22.975426926s" totalImagesPullingTime="21.780816908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:40.975169585 +0000 UTC m=+23.777256107" watchObservedRunningTime="2026-07-08 17:33:40.975426926 +0000 UTC m=+23.777513445" Jul 08 17:33:41.019854 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.019783 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pk4h7" podStartSLOduration=1.16773612 podStartE2EDuration="23.019761367s" totalImagesPullingTime="21.852025247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:41.017713963 +0000 UTC m=+23.819800482" watchObservedRunningTime="2026-07-08 17:33:41.019761367 +0000 UTC m=+23.821847887" Jul 08 17:33:41.020562 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.020525 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-65.ec2.internal" podStartSLOduration=22.020513955 podStartE2EDuration="22.020513955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:40.989761873 +0000 UTC m=+23.791848393" watchObservedRunningTime="2026-07-08 17:33:41.020513955 +0000 UTC m=+23.822600515" Jul 08 17:33:41.785137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.784904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:41.785269 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:41.785235 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:41.971231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.971160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kthzc" event={"ID":"5997a67d-d6be-44d7-be12-0c42de6ded38","Type":"ContainerStarted","Data":"ecab1a8eec8112db808808c0ee519b623ca230ddf52ca5fd1dee01e0dbca3abc"} Jul 08 17:33:41.972575 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.972549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2kz8j" event={"ID":"f80c2ab9-7f70-4fb4-a04d-6c865232a4a7","Type":"ContainerStarted","Data":"fed027789499814ef3ca1255f94209baefed8513aac64a99a5d2d05ad20b48b5"} Jul 08 17:33:41.973988 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.973964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" event={"ID":"1ee437b7-95ac-4fbb-9895-f1acb4341051","Type":"ContainerStarted","Data":"f8ff6a316c65a2e9dfe5d4532a8bd93e954c34a5d2363c64ca10bb79e6abde16"} Jul 08 17:33:41.975376 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.975349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x66br" event={"ID":"8a76373e-ad42-4a73-b723-1cb32ed2f2b7","Type":"ContainerStarted","Data":"6aea996aef37b85be7b7f977e7dacdce391bb8198dea6e8c19f4b09adb2afeac"} Jul 08 17:33:41.978067 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.978047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"49caee12ba61e9f4df69635e44d3b483df25db68211846f801fbba8acb60fe3e"} Jul 08 17:33:41.978144 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.978100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"920f56b1e5aea62b1207146ba42f9f3259d5ff1c83565a695c4ba480c82f6ab8"} Jul 08 17:33:41.978144 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.978139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"cd958e3a3b426367a1c935bf2d06b4d00fcd6f91bed55f19679118e31fad1c04"} Jul 08 17:33:41.978249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.978168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"1176d499023b2a4424bcebf9e686cd7794fa7e47302e983df43fd10259d215c6"} Jul 08 17:33:41.978249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.978204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"66b8d843c86ed4fdc889b64831f94e2e9fb035a02a6c1a7420836ae5a7d808fc"} Jul 08 17:33:41.979366 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.979347 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="9c04cd9297fa5b7870e6e259914583fb73cd165a5509b177a10a11583d75317a" exitCode=0 Jul 08 17:33:41.979454 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.979423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"9c04cd9297fa5b7870e6e259914583fb73cd165a5509b177a10a11583d75317a"} Jul 08 17:33:41.980836 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.980817 2579 generic.go:356] "Generic (PLEG): container finished" podID="aca91109daef0c5ad11720d851b32cd8" containerID="6a0f28ab65644bc26de6fa4fd3d6bd264e1598ed5588ba86a6af8818bac9ded0" exitCode=0 Jul 08 17:33:41.980901 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.980885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"aca91109daef0c5ad11720d851b32cd8","Type":"ContainerDied","Data":"6a0f28ab65644bc26de6fa4fd3d6bd264e1598ed5588ba86a6af8818bac9ded0"} Jul 08 17:33:41.987317 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:41.987274 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kthzc" podStartSLOduration=2.435331017 podStartE2EDuration="23.987257938s" totalImagesPullingTime="21.551926921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:41.986563326 +0000 UTC m=+24.788649850" watchObservedRunningTime="2026-07-08 17:33:41.987257938 +0000 UTC m=+24.789344458" Jul 08 17:33:42.045776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.045729 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x66br" podStartSLOduration=2.234689056 podStartE2EDuration="24.045710171s" totalImagesPullingTime="21.811021115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:42.045320352 +0000 UTC m=+24.847406875" watchObservedRunningTime="2026-07-08 17:33:42.045710171 +0000 UTC m=+24.847796692" Jul 08 17:33:42.060490 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.060454 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2kz8j" podStartSLOduration=10.941860354 podStartE2EDuration="24.060440447s" totalImagesPullingTime="13.118580093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:42.060290188 +0000 UTC m=+24.862376706" watchObservedRunningTime="2026-07-08 17:33:42.060440447 +0000 UTC m=+24.862526987" Jul 08 17:33:42.785779 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.785745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:42.786067 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.785792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:42.786067 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:42.785874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:42.786067 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:42.786026 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:42.984894 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.984861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" event={"ID":"aca91109daef0c5ad11720d851b32cd8","Type":"ContainerStarted","Data":"b551ee144b807406f6cfc8e02ef4c12e08528e3514f47b77d5706803fc342080"} Jul 08 17:33:42.986284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:42.986258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kbr5b" event={"ID":"f1702b59-fac0-4bdb-b651-5d08a675ee86","Type":"ContainerStarted","Data":"d8545c6c7c30c7525f7339d619f48396937525a0685534a45d06f7a453201562"} Jul 08 17:33:43.000057 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.000009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-65.ec2.internal" podStartSLOduration=23.999994492 podStartE2EDuration="23.999994492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:42.999803486 +0000 UTC m=+25.801890005" watchObservedRunningTime="2026-07-08 17:33:42.999994492 +0000 UTC m=+25.802081011" Jul 08 17:33:43.013668 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.013626 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kbr5b" podStartSLOduration=3.246279187 podStartE2EDuration="25.013614232s" totalImagesPullingTime="21.767335045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:43.013337912 +0000 UTC m=+25.815424430" watchObservedRunningTime="2026-07-08 17:33:43.013614232 +0000 UTC m=+25.815700741" Jul 08 17:33:43.659692 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.659654 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 17:33:43.691562 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.691453 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T17:33:43.65968031Z","UUID":"e945ff00-267d-45a4-b508-6ec46508230b","Handler":null,"Name":"","Endpoint":""} Jul 08 17:33:43.693284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.693207 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 17:33:43.693284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.693251 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 17:33:43.785569 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.785501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:43.785712 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:43.785684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:43.990276 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.990241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" event={"ID":"1ee437b7-95ac-4fbb-9895-f1acb4341051","Type":"ContainerStarted","Data":"bc756802fe176b5298007e6ef08b2b8482102126ae2f0843d70907f346744322"} Jul 08 17:33:43.993427 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:43.993405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"0ac16140dbdc2ae5f2991ec24b565c79d1c33d9863452e8102a599792a9f6836"} Jul 08 17:33:44.785222 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:44.785058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:44.785402 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:44.785057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:44.785402 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:44.785310 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:44.785497 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:44.785407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:45.121963 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:45.121885 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:45.122413 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:45.122397 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:45.785135 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:45.785112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:45.785269 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:45.785246 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:46.000490 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.000385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" event={"ID":"1ee437b7-95ac-4fbb-9895-f1acb4341051","Type":"ContainerStarted","Data":"d29ee86bc545cf297e8db2a0719037b511907cf8d62234ba30e3c90c4792f51a"} Jul 08 17:33:46.003973 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.003936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" event={"ID":"1bb051d6-ad34-49e8-bf5c-cb6b3fc2d1d6","Type":"ContainerStarted","Data":"8f2ccc90fafc14349027b46f27169b002628b0544b79904ebef5965e70b96b0f"} Jul 08 17:33:46.004105 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.004090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:46.004558 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.004538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x66br" Jul 08 17:33:46.020287 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.020242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7ztsd" podStartSLOduration=2.746296865 podStartE2EDuration="28.020230604s" totalImagesPullingTime="25.273933739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:46.019216887 +0000 UTC m=+28.821303406" watchObservedRunningTime="2026-07-08 17:33:46.020230604 +0000 UTC m=+28.822317122" Jul 08 17:33:46.049439 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.049387 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" podStartSLOduration=6.230216412 podStartE2EDuration="28.0493696s" totalImagesPullingTime="21.819153188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:46.04885774 +0000 UTC m=+28.850944272" watchObservedRunningTime="2026-07-08 17:33:46.0493696 +0000 UTC m=+28.851456120" Jul 08 17:33:46.785114 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.784940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:46.785855 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:46.784940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:46.785855 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:46.785175 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:46.785855 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:46.785276 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:47.007622 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.007145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:47.007622 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.007224 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:47.007622 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.007275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:47.027787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.027754 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:47.027933 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.027913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:33:47.786317 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:47.786285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:47.786841 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:47.786377 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:48.024264 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.023778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75zck"] Jul 08 17:33:48.024264 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.023900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:48.024264 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:48.023972 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:48.027872 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.027838 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7x7wz"] Jul 08 17:33:48.028003 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.027889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmqqm"] Jul 08 17:33:48.028003 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.027960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:48.028123 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:48.028046 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:48.028123 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:48.028109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:48.028223 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:48.028183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:49.785221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:49.785258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:49.785335 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:49.785350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:49.785446 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:49.785477 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:49.785467 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:50.014922 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:50.014848 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="944a51b5c3c4ba0854411f705bcc08890e004e0a09839713bbd8127f68c14061" exitCode=0 Jul 08 17:33:50.014922 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:50.014897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"944a51b5c3c4ba0854411f705bcc08890e004e0a09839713bbd8127f68c14061"} Jul 08 17:33:50.400629 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:50.400554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:50.400629 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:50.400617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:50.400833 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.400706 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:50.400833 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.400713 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:50.400833 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.400757 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret podName:4a043eb4-3ebd-42f9-b1b9-0d96dd93553a nodeName:}" failed. No retries permitted until 2026-07-08 17:34:22.400742962 +0000 UTC m=+65.202829459 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret") pod "global-pull-secret-syncer-7x7wz" (UID: "4a043eb4-3ebd-42f9-b1b9-0d96dd93553a") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:50.400833 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.400769 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs podName:d1bc455f-6a8a-4c83-aef5-022768f814f0 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:22.400763741 +0000 UTC m=+65.202850238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs") pod "network-metrics-daemon-nmqqm" (UID: "d1bc455f-6a8a-4c83-aef5-022768f814f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:50.501984 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:50.501949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:50.502153 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.502114 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:50.502153 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.502138 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:50.502153 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.502151 2579 projected.go:196] Error preparing data for projected volume kube-api-access-64cn2 for pod openshift-network-diagnostics/network-check-target-75zck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:50.502304 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:50.502209 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2 podName:e3d6ae7b-771d-485a-994d-9d5942fdd7e8 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:22.502190923 +0000 UTC m=+65.304277431 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-64cn2" (UniqueName: "kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2") pod "network-check-target-75zck" (UID: "e3d6ae7b-771d-485a-994d-9d5942fdd7e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:51.019210 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:51.019180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerStarted","Data":"9e7d5b58b3da890817b85c7157b2c17ea5fdb45b17cb278f396e2a3d32f47a58"} Jul 08 17:33:51.785640 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:51.785609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:51.785775 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:51.785620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:51.785775 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:51.785700 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:51.785853 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:51.785771 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:51.785853 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:51.785816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:51.785929 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:51.785866 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:52.023443 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:52.023412 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="9e7d5b58b3da890817b85c7157b2c17ea5fdb45b17cb278f396e2a3d32f47a58" exitCode=0 Jul 08 17:33:52.023798 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:52.023468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"9e7d5b58b3da890817b85c7157b2c17ea5fdb45b17cb278f396e2a3d32f47a58"} Jul 08 17:33:53.785694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:53.785671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:53.785694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:53.785696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:53.786165 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:53.785671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:53.786165 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:53.785805 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmqqm" podUID="d1bc455f-6a8a-4c83-aef5-022768f814f0" Jul 08 17:33:53.786165 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:53.785907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7x7wz" podUID="4a043eb4-3ebd-42f9-b1b9-0d96dd93553a" Jul 08 17:33:53.786165 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:53.785961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-75zck" podUID="e3d6ae7b-771d-485a-994d-9d5942fdd7e8" Jul 08 17:33:54.030070 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.030035 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="ed19302c5605d6896ea8b4c59d5b7123619e90a5d11a450138e918f340a8ee84" exitCode=0 Jul 08 17:33:54.030223 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.030088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"ed19302c5605d6896ea8b4c59d5b7123619e90a5d11a450138e918f340a8ee84"} Jul 08 17:33:54.489847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.489819 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-65.ec2.internal" event="NodeReady" Jul 08 17:33:54.489991 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.489919 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 17:33:54.524904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.524882 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt"] Jul 08 17:33:54.527403 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.527385 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:33:54.527542 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.527526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" Jul 08 17:33:54.529070 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.529055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wspx5"] Jul 08 17:33:54.529195 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.529180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.529797 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.529777 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.529917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.529886 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-j7gmv\"" Jul 08 17:33:54.530098 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.530082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.530640 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.530626 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cjdr8"] Jul 08 17:33:54.530753 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.530739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.531847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.531828 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n999c\"" Jul 08 17:33:54.531956 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.531937 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 17:33:54.532245 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.532232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.532329 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.532291 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 17:33:54.534974 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.534957 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 17:33:54.535213 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535195 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 17:33:54.535304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.535374 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535293 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.535374 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535312 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-r7v8g\"" Jul 08 17:33:54.535459 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535393 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h4l54\"" Jul 08 17:33:54.535761 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 17:33:54.535827 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535744 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.535887 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 17:33:54.535887 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.535816 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.537834 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.537816 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 17:33:54.543519 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.543499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt"] Jul 08 17:33:54.550708 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.550685 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wspx5"] Jul 08 17:33:54.552049 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.552022 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 17:33:54.552272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.552253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:33:54.552391 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.552358 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 17:33:54.552799 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.552783 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 17:33:54.553086 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.553063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cjdr8"] Jul 08 17:33:54.554475 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.554458 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-td4sp"] Jul 08 17:33:54.556493 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.556476 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt"] Jul 08 17:33:54.556923 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.556906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.558447 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.558430 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g"] Jul 08 17:33:54.558543 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.558529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.559047 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.559025 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 17:33:54.559047 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.559047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 17:33:54.559191 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.559094 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p74p\"" Jul 08 17:33:54.559983 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.559966 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59c88ffcb5-g8nl9"] Jul 08 17:33:54.560110 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.560093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.560760 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.560745 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.561286 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.561272 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 17:33:54.561713 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.561683 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg"] Jul 08 17:33:54.561793 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.561764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.562241 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562223 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8t799\"" Jul 08 17:33:54.562337 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562321 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 17:33:54.562481 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562463 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.562636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 17:33:54.562636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562557 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-chmsj\"" Jul 08 17:33:54.562821 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562808 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.562877 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.562849 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.563050 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.563038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 17:33:54.563868 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.563850 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zp2d5\"" Jul 08 17:33:54.563974 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.563951 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gmzgc"] Jul 08 17:33:54.564267 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.564141 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.565462 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565441 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.565549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 17:33:54.565549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565485 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 17:33:54.565549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 17:33:54.565706 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565685 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.565942 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.565923 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 17:33:54.566073 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.566059 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-h7whl"] Jul 08 17:33:54.566164 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.566151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.567516 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.567501 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86"] Jul 08 17:33:54.567624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.567615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.569066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.569049 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7"] Jul 08 17:33:54.569163 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.569148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" Jul 08 17:33:54.570646 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.570623 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 17:33:54.570767 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.570679 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6gqv2\"" Jul 08 17:33:54.570985 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.570950 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.571116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.571097 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.571189 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.571174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 17:33:54.571536 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.571516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.572046 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-td4sp"] Jul 08 17:33:54.572476 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572458 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.572563 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 17:33:54.572563 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572508 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.572691 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572590 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wnq9l\"" Jul 08 17:33:54.572691 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572661 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.572691 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572670 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f4g57\"" Jul 08 17:33:54.572828 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572772 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 17:33:54.572828 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572777 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 17:33:54.572966 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.572840 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8242s\"" Jul 08 17:33:54.574094 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.573145 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.577199 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.575169 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:54.577199 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.575888 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l79fh\"" Jul 08 17:33:54.577199 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.576429 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 17:33:54.577199 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.576544 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:54.577199 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.576433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g"] Jul 08 17:33:54.577407 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.577314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt"] Jul 08 17:33:54.589582 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.580331 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-h7whl"] Jul 08 17:33:54.589582 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.580372 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7"] Jul 08 17:33:54.589582 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.580397 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg"] Jul 08 17:33:54.589582 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.581632 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59c88ffcb5-g8nl9"] Jul 08 17:33:54.607026 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.606996 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86"] Jul 08 17:33:54.608705 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.608681 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmzgc"] Jul 08 17:33:54.632185 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632286 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.632286 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632286 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-snapshots\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.632411 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632411 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-config\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.632519 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-trusted-ca\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.632519 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c32f2368-e81f-42ea-804f-62c5d495c6a8-serving-cert\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.632519 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj76\" (UniqueName: \"kubernetes.io/projected/c32f2368-e81f-42ea-804f-62c5d495c6a8-kube-api-access-4nj76\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20bcd1ae-4584-4f6a-a8ec-90982f45475d-serving-cert\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtk7\" (UniqueName: \"kubernetes.io/projected/20bcd1ae-4584-4f6a-a8ec-90982f45475d-kube-api-access-bxtk7\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.632689 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2lp\" (UniqueName: \"kubernetes.io/projected/defd5d3c-19fc-47a3-9e7e-08a5e04c17e3-kube-api-access-7l2lp\") pod \"volume-data-source-validator-5d456fdbcd-pbmdt\" (UID: \"defd5d3c-19fc-47a3-9e7e-08a5e04c17e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" Jul 08 17:33:54.632961 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z87j\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.632961 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.632745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-tmp\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.733717 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-default-certificate\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1238bd9b-fa90-4059-89ff-f57876652cde-config\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjqk\" (UniqueName: \"kubernetes.io/projected/48591443-82cb-4929-b438-2e2f8768bfcd-kube-api-access-chjqk\") pod \"network-check-source-989f8b8dc-tnw86\" (UID: \"48591443-82cb-4929-b438-2e2f8768bfcd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c4509a-bf4d-423b-82d1-72d7f666a32f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2lp\" (UniqueName: \"kubernetes.io/projected/defd5d3c-19fc-47a3-9e7e-08a5e04c17e3-kube-api-access-7l2lp\") pod \"volume-data-source-validator-5d456fdbcd-pbmdt\" (UID: \"defd5d3c-19fc-47a3-9e7e-08a5e04c17e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0c4509a-bf4d-423b-82d1-72d7f666a32f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.733936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z87j\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.733980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj59t\" (UniqueName: \"kubernetes.io/projected/7d531774-6047-4988-9880-c8d5044bfd0d-kube-api-access-hj59t\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-config\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3651a18-f091-4b86-8653-8b1f07453339-tmp-dir\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.734202 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhnjq\" (UniqueName: \"kubernetes.io/projected/d0c4509a-bf4d-423b-82d1-72d7f666a32f-kube-api-access-hhnjq\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdc4\" (UniqueName: \"kubernetes.io/projected/e5d590a0-e899-485c-901e-088c7bdabc9a-kube-api-access-lzdc4\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfjf\" (UniqueName: \"kubernetes.io/projected/b3651a18-f091-4b86-8653-8b1f07453339-kube-api-access-vpfjf\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj76\" (UniqueName: \"kubernetes.io/projected/c32f2368-e81f-42ea-804f-62c5d495c6a8-kube-api-access-4nj76\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3651a18-f091-4b86-8653-8b1f07453339-config-volume\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20bcd1ae-4584-4f6a-a8ec-90982f45475d-serving-cert\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/e853ca21-cbdf-4916-bd5e-6755ea765ac7-kube-api-access-qb7lt\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5s2c\" (UniqueName: \"kubernetes.io/projected/0856353e-c7bc-4d49-ae1c-f89082b72417-kube-api-access-h5s2c\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d531774-6047-4988-9880-c8d5044bfd0d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq6nd\" (UniqueName: \"kubernetes.io/projected/1238bd9b-fa90-4059-89ff-f57876652cde-kube-api-access-fq6nd\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-stats-auth\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.734624 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-snapshots\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-trusted-ca\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c32f2368-e81f-42ea-804f-62c5d495c6a8-serving-cert\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1238bd9b-fa90-4059-89ff-f57876652cde-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-config\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d590a0-e899-485c-901e-088c7bdabc9a-cert\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734924 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.734937 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734943 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-tmp\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.735250 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.734982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtk7\" (UniqueName: \"kubernetes.io/projected/20bcd1ae-4584-4f6a-a8ec-90982f45475d-kube-api-access-bxtk7\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-snapshots\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735733 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c32f2368-e81f-42ea-804f-62c5d495c6a8-tmp\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735924 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c32f2368-e81f-42ea-804f-62c5d495c6a8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.735982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20bcd1ae-4584-4f6a-a8ec-90982f45475d-trusted-ca\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.736009 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.734951 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:33:54.736360 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.736058 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.236042704 +0000 UTC m=+38.038129200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:33:54.736360 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.736221 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.739707 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.739683 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.739883 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.739860 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.740175 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.740152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20bcd1ae-4584-4f6a-a8ec-90982f45475d-serving-cert\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.742134 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.742012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c32f2368-e81f-42ea-804f-62c5d495c6a8-serving-cert\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.744517 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.744491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.745087 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.745062 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z87j\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:54.745529 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.745508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtk7\" (UniqueName: \"kubernetes.io/projected/20bcd1ae-4584-4f6a-a8ec-90982f45475d-kube-api-access-bxtk7\") pod \"console-operator-68b757fb67-wspx5\" (UID: \"20bcd1ae-4584-4f6a-a8ec-90982f45475d\") " pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.745897 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.745873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2lp\" (UniqueName: \"kubernetes.io/projected/defd5d3c-19fc-47a3-9e7e-08a5e04c17e3-kube-api-access-7l2lp\") pod \"volume-data-source-validator-5d456fdbcd-pbmdt\" (UID: \"defd5d3c-19fc-47a3-9e7e-08a5e04c17e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" Jul 08 17:33:54.745978 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.745962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj76\" (UniqueName: \"kubernetes.io/projected/c32f2368-e81f-42ea-804f-62c5d495c6a8-kube-api-access-4nj76\") pod \"insights-operator-5c847dcfc4-cjdr8\" (UID: \"c32f2368-e81f-42ea-804f-62c5d495c6a8\") " pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.835952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-default-certificate\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.835952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1238bd9b-fa90-4059-89ff-f57876652cde-config\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.835952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjqk\" (UniqueName: \"kubernetes.io/projected/48591443-82cb-4929-b438-2e2f8768bfcd-kube-api-access-chjqk\") pod \"network-check-source-989f8b8dc-tnw86\" (UID: \"48591443-82cb-4929-b438-2e2f8768bfcd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" Jul 08 17:33:54.835952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c4509a-bf4d-423b-82d1-72d7f666a32f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0c4509a-bf4d-423b-82d1-72d7f666a32f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.835997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj59t\" (UniqueName: \"kubernetes.io/projected/7d531774-6047-4988-9880-c8d5044bfd0d-kube-api-access-hj59t\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3651a18-f091-4b86-8653-8b1f07453339-tmp-dir\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836136 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.336116434 +0000 UTC m=+38.138202934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836175 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhnjq\" (UniqueName: \"kubernetes.io/projected/d0c4509a-bf4d-423b-82d1-72d7f666a32f-kube-api-access-hhnjq\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdc4\" (UniqueName: \"kubernetes.io/projected/e5d590a0-e899-485c-901e-088c7bdabc9a-kube-api-access-lzdc4\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836231 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.336214749 +0000 UTC m=+38.138301261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfjf\" (UniqueName: \"kubernetes.io/projected/b3651a18-f091-4b86-8653-8b1f07453339-kube-api-access-vpfjf\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3651a18-f091-4b86-8653-8b1f07453339-config-volume\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.836787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/e853ca21-cbdf-4916-bd5e-6755ea765ac7-kube-api-access-qb7lt\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5s2c\" (UniqueName: \"kubernetes.io/projected/0856353e-c7bc-4d49-ae1c-f89082b72417-kube-api-access-h5s2c\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d531774-6047-4988-9880-c8d5044bfd0d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq6nd\" (UniqueName: \"kubernetes.io/projected/1238bd9b-fa90-4059-89ff-f57876652cde-kube-api-access-fq6nd\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-stats-auth\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3651a18-f091-4b86-8653-8b1f07453339-tmp-dir\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836514 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836524 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1238bd9b-fa90-4059-89ff-f57876652cde-config\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.336564166 +0000 UTC m=+38.138650667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836648 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1238bd9b-fa90-4059-89ff-f57876652cde-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836682 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.336670994 +0000 UTC m=+38.138757497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d590a0-e899-485c-901e-088c7bdabc9a-cert\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.836731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.837465 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836822 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.836857 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.33684643 +0000 UTC m=+38.138932927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.837287 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.837339 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3651a18-f091-4b86-8653-8b1f07453339-config-volume\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.837377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c4509a-bf4d-423b-82d1-72d7f666a32f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.837420 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.837492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:54.837548 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:33:55.337532204 +0000 UTC m=+38.139618718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:33:54.838313 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.837648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d531774-6047-4988-9880-c8d5044bfd0d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.838727 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.838696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0c4509a-bf4d-423b-82d1-72d7f666a32f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.839284 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.839261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1238bd9b-fa90-4059-89ff-f57876652cde-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.839395 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.839332 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-default-certificate\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.839892 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.839873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5d590a0-e899-485c-901e-088c7bdabc9a-cert\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.840243 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.840223 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-stats-auth\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.851724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjqk\" (UniqueName: \"kubernetes.io/projected/48591443-82cb-4929-b438-2e2f8768bfcd-kube-api-access-chjqk\") pod \"network-check-source-989f8b8dc-tnw86\" (UID: \"48591443-82cb-4929-b438-2e2f8768bfcd\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.852265 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdc4\" (UniqueName: \"kubernetes.io/projected/e5d590a0-e899-485c-901e-088c7bdabc9a-kube-api-access-lzdc4\") pod \"ingress-canary-gmzgc\" (UID: \"e5d590a0-e899-485c-901e-088c7bdabc9a\") " pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.853720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhnjq\" (UniqueName: \"kubernetes.io/projected/d0c4509a-bf4d-423b-82d1-72d7f666a32f-kube-api-access-hhnjq\") pod \"kube-storage-version-migrator-operator-846bf668f7-qrdlt\" (UID: \"d0c4509a-bf4d-423b-82d1-72d7f666a32f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.854104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj59t\" (UniqueName: \"kubernetes.io/projected/7d531774-6047-4988-9880-c8d5044bfd0d-kube-api-access-hj59t\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.855099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/e853ca21-cbdf-4916-bd5e-6755ea765ac7-kube-api-access-qb7lt\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.855158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfjf\" (UniqueName: \"kubernetes.io/projected/b3651a18-f091-4b86-8653-8b1f07453339-kube-api-access-vpfjf\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:54.858132 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.855665 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5s2c\" (UniqueName: \"kubernetes.io/projected/0856353e-c7bc-4d49-ae1c-f89082b72417-kube-api-access-h5s2c\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:54.860062 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.860039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq6nd\" (UniqueName: \"kubernetes.io/projected/1238bd9b-fa90-4059-89ff-f57876652cde-kube-api-access-fq6nd\") pod \"service-ca-operator-5d46d9ffd4-l82dg\" (UID: \"1238bd9b-fa90-4059-89ff-f57876652cde\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.860160 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.860151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:33:54.867803 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.867772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" Jul 08 17:33:54.908405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.907204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" Jul 08 17:33:54.933362 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.932906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" Jul 08 17:33:54.940665 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.940638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gmzgc" Jul 08 17:33:54.955688 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:54.955664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" Jul 08 17:33:55.036407 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.034279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt"] Jul 08 17:33:55.040123 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.040083 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefd5d3c_19fc_47a3_9e7e_08a5e04c17e3.slice/crio-e2edd02edab3296bfdbf85b5be14b67a66df0c54a120e6d4d0a03027aaed4ba6 WatchSource:0}: Error finding container e2edd02edab3296bfdbf85b5be14b67a66df0c54a120e6d4d0a03027aaed4ba6: Status 404 returned error can't find the container with id e2edd02edab3296bfdbf85b5be14b67a66df0c54a120e6d4d0a03027aaed4ba6 Jul 08 17:33:55.052874 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.051663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wspx5"] Jul 08 17:33:55.060139 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.060110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-cjdr8"] Jul 08 17:33:55.063759 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.062787 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bcd1ae_4584_4f6a_a8ec_90982f45475d.slice/crio-55e2059758f152dcd4c554bed8769fee8907add6005eeaf730bd4ddf979ad960 WatchSource:0}: Error finding container 55e2059758f152dcd4c554bed8769fee8907add6005eeaf730bd4ddf979ad960: Status 404 returned error can't find the container with id 55e2059758f152dcd4c554bed8769fee8907add6005eeaf730bd4ddf979ad960 Jul 08 17:33:55.107239 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.107184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:55.139272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.139244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt"] Jul 08 17:33:55.145135 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.145112 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c4509a_bf4d_423b_82d1_72d7f666a32f.slice/crio-c54cbd40768f9f2356bc04b975cbf69e8dd564d9e7ec0c276c98bb510287034e WatchSource:0}: Error finding container c54cbd40768f9f2356bc04b975cbf69e8dd564d9e7ec0c276c98bb510287034e: Status 404 returned error can't find the container with id c54cbd40768f9f2356bc04b975cbf69e8dd564d9e7ec0c276c98bb510287034e Jul 08 17:33:55.154875 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.154854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg"] Jul 08 17:33:55.158047 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.158027 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1238bd9b_fa90_4059_89ff_f57876652cde.slice/crio-ddef26333e4789d10b52f3b14fa5022f287138dd15aec6db5e96b7911beba355 WatchSource:0}: Error finding container ddef26333e4789d10b52f3b14fa5022f287138dd15aec6db5e96b7911beba355: Status 404 returned error can't find the container with id ddef26333e4789d10b52f3b14fa5022f287138dd15aec6db5e96b7911beba355 Jul 08 17:33:55.164581 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.164562 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86"] Jul 08 17:33:55.167082 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.167064 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48591443_82cb_4929_b438_2e2f8768bfcd.slice/crio-7a8a5be5e1cf5ccfa3827a020b0366478586ed79f0c00098256d2f97ae5525cc WatchSource:0}: Error finding container 7a8a5be5e1cf5ccfa3827a020b0366478586ed79f0c00098256d2f97ae5525cc: Status 404 returned error can't find the container with id 7a8a5be5e1cf5ccfa3827a020b0366478586ed79f0c00098256d2f97ae5525cc Jul 08 17:33:55.169002 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.168984 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gmzgc"] Jul 08 17:33:55.172254 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:33:55.172233 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d590a0_e899_485c_901e_088c7bdabc9a.slice/crio-2c74846060d9b682e6ce6c358da73087912110f45df1450ba145cddbaa86f639 WatchSource:0}: Error finding container 2c74846060d9b682e6ce6c358da73087912110f45df1450ba145cddbaa86f639: Status 404 returned error can't find the container with id 2c74846060d9b682e6ce6c358da73087912110f45df1450ba145cddbaa86f639 Jul 08 17:33:55.209032 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.209005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:55.240366 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.240345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:55.240471 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.240461 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:33:55.240517 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.240476 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:33:55.240569 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.240518 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.240506117 +0000 UTC m=+39.042592613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:33:55.341173 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:55.341173 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341141 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:55.341333 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341253 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:33:55.341333 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341303 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:33:55.341429 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341304 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.341291522 +0000 UTC m=+39.143378018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:33:55.341429 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:55.341551 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341437 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.341422703 +0000 UTC m=+39.143509214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:33:55.341551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:55.341551 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341488 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:55.341551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:55.341551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.341541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341556 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.34154625 +0000 UTC m=+39.143632746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341579 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341616 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341641 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.341628871 +0000 UTC m=+39.143715384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341658 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.341647907 +0000 UTC m=+39.143734422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:33:55.341789 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:55.341671 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:33:56.341666312 +0000 UTC m=+39.143752809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:33:55.785345 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.785311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:33:55.785810 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.785795 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:33:55.786229 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.786211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:33:55.788973 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.788950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktdt9\"" Jul 08 17:33:55.789082 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.788992 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:33:55.789082 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.788950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jhm\"" Jul 08 17:33:55.789240 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:55.789224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:33:56.041292 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.041163 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" event={"ID":"20bcd1ae-4584-4f6a-a8ec-90982f45475d","Type":"ContainerStarted","Data":"55e2059758f152dcd4c554bed8769fee8907add6005eeaf730bd4ddf979ad960"} Jul 08 17:33:56.043007 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.042979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" event={"ID":"48591443-82cb-4929-b438-2e2f8768bfcd","Type":"ContainerStarted","Data":"7a8a5be5e1cf5ccfa3827a020b0366478586ed79f0c00098256d2f97ae5525cc"} Jul 08 17:33:56.046657 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.046631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" event={"ID":"d0c4509a-bf4d-423b-82d1-72d7f666a32f","Type":"ContainerStarted","Data":"c54cbd40768f9f2356bc04b975cbf69e8dd564d9e7ec0c276c98bb510287034e"} Jul 08 17:33:56.050008 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.049946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" event={"ID":"c32f2368-e81f-42ea-804f-62c5d495c6a8","Type":"ContainerStarted","Data":"bbed129c40be42dbf651670d69798626d944be10ed47c864437af4426381f1be"} Jul 08 17:33:56.052015 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.051989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" event={"ID":"defd5d3c-19fc-47a3-9e7e-08a5e04c17e3","Type":"ContainerStarted","Data":"e2edd02edab3296bfdbf85b5be14b67a66df0c54a120e6d4d0a03027aaed4ba6"} Jul 08 17:33:56.054444 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.054419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmzgc" event={"ID":"e5d590a0-e899-485c-901e-088c7bdabc9a","Type":"ContainerStarted","Data":"2c74846060d9b682e6ce6c358da73087912110f45df1450ba145cddbaa86f639"} Jul 08 17:33:56.056338 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.056305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" event={"ID":"1238bd9b-fa90-4059-89ff-f57876652cde","Type":"ContainerStarted","Data":"ddef26333e4789d10b52f3b14fa5022f287138dd15aec6db5e96b7911beba355"} Jul 08 17:33:56.249398 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.248742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:56.249398 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.249054 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:33:56.249398 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.249068 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:33:56.249398 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.249125 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.24910811 +0000 UTC m=+41.051194609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:56.349842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:56.349819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.349982 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350042 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.350024837 +0000 UTC m=+41.152111356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350437 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350489 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.3504735 +0000 UTC m=+41.152560000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350545 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.350565138 +0000 UTC m=+41.152651634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350647 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350678 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.350667439 +0000 UTC m=+41.152753943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350734 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.350724408 +0000 UTC m=+41.152810906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350799 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:33:56.351339 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:56.350836 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:58.350815788 +0000 UTC m=+41.152902288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:33:58.026394 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.026123 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:58.071039 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.070993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" event={"ID":"defd5d3c-19fc-47a3-9e7e-08a5e04c17e3","Type":"ContainerStarted","Data":"20ebc20fcdf4d999ce41d7e1b0c3c09c7b32ea87b77b376a3ff0f31b94a33e75"} Jul 08 17:33:58.091004 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.090956 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-pbmdt" podStartSLOduration=26.100672764 podStartE2EDuration="28.090937224s" totalImagesPullingTime="1.99026446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:58.089722843 +0000 UTC m=+40.891809361" watchObservedRunningTime="2026-07-08 17:33:58.090937224 +0000 UTC m=+40.893023742" Jul 08 17:33:58.102079 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.101996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:58.197871 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.197828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:58.274373 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.274340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:33:58.274664 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.274631 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:33:58.274761 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.274671 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:33:58.274761 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.274732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.274713077 +0000 UTC m=+45.076799596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:33:58.296827 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.296760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:33:58.375651 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:58.375829 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:33:58.375829 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:33:58.375944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:33:58.375944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:33:58.376037 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:33:58.375954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:33:58.376088 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376071 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:33:58.376187 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376118 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376104589 +0000 UTC m=+45.178191088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:33:58.376480 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376443078 +0000 UTC m=+45.178529575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:33:58.376587 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376540 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:33:58.376587 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376573 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376562623 +0000 UTC m=+45.178649132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:33:58.376743 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376624 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:33:58.376743 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376658 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376645623 +0000 UTC m=+45.178732120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:33:58.376743 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376707 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:58.376743 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376737 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376726968 +0000 UTC m=+45.178813465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:33:58.376931 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376784 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:33:58.376931 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:33:58.376813 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:34:02.376803269 +0000 UTC m=+45.178889770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:34:02.309715 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.309675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:02.310179 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.309852 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:34:02.310179 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.309887 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:34:02.310179 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.309959 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.309937951 +0000 UTC m=+53.112024449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:34:02.410671 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:34:02.410862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:34:02.410862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:34:02.410862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:02.410862 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.410809 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:34:02.410862 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.410857 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.410895 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.410862 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.410845764 +0000 UTC m=+53.212932260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.410933 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.41091513 +0000 UTC m=+53.213001631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:02.410999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411014 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411046 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.411034955 +0000 UTC m=+53.213121451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411067 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.411060148 +0000 UTC m=+53.213146644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411078 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.411073205 +0000 UTC m=+53.213159701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:34:02.411107 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411086 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:34:02.411522 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:02.411128 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:10.411116596 +0000 UTC m=+53.213203136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:34:05.090961 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.090722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" event={"ID":"d0c4509a-bf4d-423b-82d1-72d7f666a32f","Type":"ContainerStarted","Data":"91baba7baa33ff7f7d23eb7f11fa2fb161c726d8e041becf2e82ae7bbee07d64"} Jul 08 17:34:05.092920 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.092674 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" event={"ID":"c32f2368-e81f-42ea-804f-62c5d495c6a8","Type":"ContainerStarted","Data":"51367225fff693904bb251f338e91c9905b6909fdd1263516c3df96c7b556c23"} Jul 08 17:34:05.096823 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.096802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerStarted","Data":"178a162c90511c998bb324fc7b40e5c24e75a5733abaa4e3283e55cc59b3db74"} Jul 08 17:34:05.098984 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.098722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gmzgc" event={"ID":"e5d590a0-e899-485c-901e-088c7bdabc9a","Type":"ContainerStarted","Data":"ea8daed55ff68c4d1e31461f9e84b085ba591acc50961b388e70d7f40e528406"} Jul 08 17:34:05.100728 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.100504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" event={"ID":"1238bd9b-fa90-4059-89ff-f57876652cde","Type":"ContainerStarted","Data":"0e6eeb09aa5e9a1d43f952c5e378a7c28bd2198fe10387acd728db6451ca5e57"} Jul 08 17:34:05.102902 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.102538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" event={"ID":"20bcd1ae-4584-4f6a-a8ec-90982f45475d","Type":"ContainerStarted","Data":"d44cf01b6be73f58977a034ea189f18bcb3939b418623746662d08bed749ddf7"} Jul 08 17:34:05.102902 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.102839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:34:05.104001 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.103870 2579 patch_prober.go:28] interesting pod/console-operator-68b757fb67-wspx5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Jul 08 17:34:05.104001 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.103921 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podUID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Jul 08 17:34:05.104206 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.104146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" event={"ID":"48591443-82cb-4929-b438-2e2f8768bfcd","Type":"ContainerStarted","Data":"f5f6bfad159ab1f7e25e4e9b2d1d8c4146c8030bab79c77130c1381b02aaf600"} Jul 08 17:34:05.109536 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.109391 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" podStartSLOduration=25.694087885 podStartE2EDuration="35.109374749s" totalImagesPullingTime="9.415286864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.108232087 +0000 UTC m=+47.910318625" watchObservedRunningTime="2026-07-08 17:34:05.109374749 +0000 UTC m=+47.911461268" Jul 08 17:34:05.156891 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.155719 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podStartSLOduration=25.62769241 podStartE2EDuration="35.15569995s" totalImagesPullingTime="9.52800754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.150914011 +0000 UTC m=+47.953000522" watchObservedRunningTime="2026-07-08 17:34:05.15569995 +0000 UTC m=+47.957786468" Jul 08 17:34:05.156891 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.156044 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-tnw86" podStartSLOduration=25.428247929 podStartE2EDuration="35.156037284s" totalImagesPullingTime="9.727789355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.129896419 +0000 UTC m=+47.931982939" watchObservedRunningTime="2026-07-08 17:34:05.156037284 +0000 UTC m=+47.958123799" Jul 08 17:34:05.170622 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.170253 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" podStartSLOduration=25.73246327 podStartE2EDuration="35.1702342s" totalImagesPullingTime="9.43777093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.168129436 +0000 UTC m=+47.970215953" watchObservedRunningTime="2026-07-08 17:34:05.1702342 +0000 UTC m=+47.972320722" Jul 08 17:34:05.205826 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.205780 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" podStartSLOduration=25.681755065 podStartE2EDuration="35.205760785s" totalImagesPullingTime="9.52400572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.205069162 +0000 UTC m=+48.007155680" watchObservedRunningTime="2026-07-08 17:34:05.205760785 +0000 UTC m=+48.007847305" Jul 08 17:34:05.222339 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.220479 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gmzgc" podStartSLOduration=1.796912377 podStartE2EDuration="11.220464631s" totalImagesPullingTime="9.423552254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:05.219071642 +0000 UTC m=+48.021158161" watchObservedRunningTime="2026-07-08 17:34:05.220464631 +0000 UTC m=+48.022551188" Jul 08 17:34:05.717218 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.717184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:05.730346 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.730311 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l"] Jul 08 17:34:05.745967 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.745941 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l"] Jul 08 17:34:05.746123 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.746042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" Jul 08 17:34:05.749237 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.749207 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 17:34:05.749381 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.749297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-59cx6\"" Jul 08 17:34:05.749861 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.749841 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 17:34:05.826103 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.826068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:05.840551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.840521 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tn2xt"] Jul 08 17:34:05.844466 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.844442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bsq\" (UniqueName: \"kubernetes.io/projected/3bf6faff-4e8f-436a-92b9-8fb3411f3a06-kube-api-access-z4bsq\") pod \"migrator-d57d58cc9-8sz8l\" (UID: \"3bf6faff-4e8f-436a-92b9-8fb3411f3a06\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" Jul 08 17:34:05.858575 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.858533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.861101 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.861075 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tn2xt"] Jul 08 17:34:05.861360 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.861340 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 17:34:05.861465 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.861346 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9pdx\"" Jul 08 17:34:05.861543 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.861466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 17:34:05.892752 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.892707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:05.945609 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c6223a6-2690-4fae-8e29-692c0118f36e-crio-socket\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.945738 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.945738 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c6223a6-2690-4fae-8e29-692c0118f36e-data-volume\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.945842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.945842 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bsq\" (UniqueName: \"kubernetes.io/projected/3bf6faff-4e8f-436a-92b9-8fb3411f3a06-kube-api-access-z4bsq\") pod \"migrator-d57d58cc9-8sz8l\" (UID: \"3bf6faff-4e8f-436a-92b9-8fb3411f3a06\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" Jul 08 17:34:05.945945 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.945854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx6q\" (UniqueName: \"kubernetes.io/projected/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-api-access-sdx6q\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:05.955529 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:05.955499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bsq\" (UniqueName: \"kubernetes.io/projected/3bf6faff-4e8f-436a-92b9-8fb3411f3a06-kube-api-access-z4bsq\") pod \"migrator-d57d58cc9-8sz8l\" (UID: \"3bf6faff-4e8f-436a-92b9-8fb3411f3a06\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" Jul 08 17:34:06.046860 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.046832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c6223a6-2690-4fae-8e29-692c0118f36e-data-volume\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.046958 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.046879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.046958 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.046937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx6q\" (UniqueName: \"kubernetes.io/projected/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-api-access-sdx6q\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.047105 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.047072 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:34:06.047213 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.047106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c6223a6-2690-4fae-8e29-692c0118f36e-data-volume\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.047213 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.047135 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:06.547119228 +0000 UTC m=+49.349205746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : secret "insights-runtime-extractor-tls" not found Jul 08 17:34:06.047213 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.047202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c6223a6-2690-4fae-8e29-692c0118f36e-crio-socket\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.047369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.047220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.047369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.047279 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c6223a6-2690-4fae-8e29-692c0118f36e-crio-socket\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.047369 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.047297 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:34:06.047369 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.047336 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:06.547325307 +0000 UTC m=+49.349411808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : configmap "kube-rbac-proxy" not found Jul 08 17:34:06.055656 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.055635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" Jul 08 17:34:06.055908 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.055893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx6q\" (UniqueName: \"kubernetes.io/projected/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-api-access-sdx6q\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.109218 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.109195 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/0.log" Jul 08 17:34:06.109562 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.109235 2579 generic.go:356] "Generic (PLEG): container finished" podID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" containerID="d44cf01b6be73f58977a034ea189f18bcb3939b418623746662d08bed749ddf7" exitCode=255 Jul 08 17:34:06.109562 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.109293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" event={"ID":"20bcd1ae-4584-4f6a-a8ec-90982f45475d","Type":"ContainerDied","Data":"d44cf01b6be73f58977a034ea189f18bcb3939b418623746662d08bed749ddf7"} Jul 08 17:34:06.109697 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.109562 2579 scope.go:117] "RemoveContainer" containerID="d44cf01b6be73f58977a034ea189f18bcb3939b418623746662d08bed749ddf7" Jul 08 17:34:06.113656 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.113136 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="178a162c90511c998bb324fc7b40e5c24e75a5733abaa4e3283e55cc59b3db74" exitCode=0 Jul 08 17:34:06.113656 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.113262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"178a162c90511c998bb324fc7b40e5c24e75a5733abaa4e3283e55cc59b3db74"} Jul 08 17:34:06.244929 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.244903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l"] Jul 08 17:34:06.247421 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:06.247386 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf6faff_4e8f_436a_92b9_8fb3411f3a06.slice/crio-bdac65d97e4bc8b18697cc39a31c4c061676f64971d5f4044adcb71308f437e9 WatchSource:0}: Error finding container bdac65d97e4bc8b18697cc39a31c4c061676f64971d5f4044adcb71308f437e9: Status 404 returned error can't find the container with id bdac65d97e4bc8b18697cc39a31c4c061676f64971d5f4044adcb71308f437e9 Jul 08 17:34:06.555176 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.555098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.555314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.555213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:06.555314 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.555247 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:34:06.555314 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.555310 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:34:06.555410 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.555312 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:07.555296914 +0000 UTC m=+50.357383411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : secret "insights-runtime-extractor-tls" not found Jul 08 17:34:06.555410 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:06.555357 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:07.555345505 +0000 UTC m=+50.357432001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : configmap "kube-rbac-proxy" not found Jul 08 17:34:06.617255 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.617224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:06.716031 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:06.716000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:07.117422 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.117392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" event={"ID":"3bf6faff-4e8f-436a-92b9-8fb3411f3a06","Type":"ContainerStarted","Data":"bdac65d97e4bc8b18697cc39a31c4c061676f64971d5f4044adcb71308f437e9"} Jul 08 17:34:07.120281 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.120256 2579 generic.go:356] "Generic (PLEG): container finished" podID="2170557d-59b8-4c8a-8575-d9ca35586250" containerID="ef2289c95c494a03fc734d566197965f0b79787a41f18e2a0fa6f3aa57108d67" exitCode=0 Jul 08 17:34:07.120404 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.120348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerDied","Data":"ef2289c95c494a03fc734d566197965f0b79787a41f18e2a0fa6f3aa57108d67"} Jul 08 17:34:07.121843 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.121828 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:34:07.122216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.122201 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/0.log" Jul 08 17:34:07.122282 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.122257 2579 generic.go:356] "Generic (PLEG): container finished" podID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" exitCode=255 Jul 08 17:34:07.122328 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.122282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" event={"ID":"20bcd1ae-4584-4f6a-a8ec-90982f45475d","Type":"ContainerDied","Data":"0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1"} Jul 08 17:34:07.122328 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.122317 2579 scope.go:117] "RemoveContainer" containerID="d44cf01b6be73f58977a034ea189f18bcb3939b418623746662d08bed749ddf7" Jul 08 17:34:07.122503 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.122484 2579 scope.go:117] "RemoveContainer" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" Jul 08 17:34:07.122697 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:07.122665 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wspx5_openshift-console-operator(20bcd1ae-4584-4f6a-a8ec-90982f45475d)\"" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podUID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" Jul 08 17:34:07.303756 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.303731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:07.405586 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.405514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:07.561976 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.561948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:07.562096 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.562047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:07.562096 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:07.562085 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:34:07.562169 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:07.562140 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:34:07.562198 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:07.562143 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:09.562128997 +0000 UTC m=+52.364215494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : secret "insights-runtime-extractor-tls" not found Jul 08 17:34:07.562198 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:07.562186 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:09.56217472 +0000 UTC m=+52.364261220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : configmap "kube-rbac-proxy" not found Jul 08 17:34:07.829296 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:07.829271 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kthzc_5997a67d-d6be-44d7-be12-0c42de6ded38/dns-node-resolver/0.log" Jul 08 17:34:08.127456 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.127383 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:34:08.127894 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.127769 2579 scope.go:117] "RemoveContainer" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" Jul 08 17:34:08.127977 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:08.127948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wspx5_openshift-console-operator(20bcd1ae-4584-4f6a-a8ec-90982f45475d)\"" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podUID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" Jul 08 17:34:08.131415 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.131391 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" event={"ID":"2170557d-59b8-4c8a-8575-d9ca35586250","Type":"ContainerStarted","Data":"12c582a6d2a81c6bcb0da6f7effb3f9afd43bb583d4ab5a06310c96bfb064282"} Jul 08 17:34:08.171280 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.171239 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qk9v6" podStartSLOduration=7.622024245 podStartE2EDuration="50.171225488s" totalImagesPullingTime="42.549201243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:08.170063979 +0000 UTC m=+50.972150524" watchObservedRunningTime="2026-07-08 17:34:08.171225488 +0000 UTC m=+50.973312040" Jul 08 17:34:08.630724 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.630688 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kz8j_f80c2ab9-7f70-4fb4-a04d-6c865232a4a7/node-ca/0.log" Jul 08 17:34:08.902682 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.902618 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rc6hj"] Jul 08 17:34:08.919801 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.919777 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rc6hj"] Jul 08 17:34:08.919931 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.919863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:08.922344 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.922312 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 17:34:08.922512 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.922498 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 17:34:08.922578 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.922564 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 17:34:08.923430 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.923415 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5ljj2\"" Jul 08 17:34:08.923472 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.923436 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 17:34:08.975297 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.975276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/775929bf-b313-4929-80f1-99005afe292e-signing-key\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:08.975429 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.975378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8h7\" (UniqueName: \"kubernetes.io/projected/775929bf-b313-4929-80f1-99005afe292e-kube-api-access-cv8h7\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:08.975490 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:08.975447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/775929bf-b313-4929-80f1-99005afe292e-signing-cabundle\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.076756 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.076728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/775929bf-b313-4929-80f1-99005afe292e-signing-key\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.076862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.076811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8h7\" (UniqueName: \"kubernetes.io/projected/775929bf-b313-4929-80f1-99005afe292e-kube-api-access-cv8h7\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.076862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.076847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/775929bf-b313-4929-80f1-99005afe292e-signing-cabundle\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.077399 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.077380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/775929bf-b313-4929-80f1-99005afe292e-signing-cabundle\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.079074 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.079056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/775929bf-b313-4929-80f1-99005afe292e-signing-key\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.085414 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.085394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8h7\" (UniqueName: \"kubernetes.io/projected/775929bf-b313-4929-80f1-99005afe292e-kube-api-access-cv8h7\") pod \"service-ca-758f599c5-rc6hj\" (UID: \"775929bf-b313-4929-80f1-99005afe292e\") " pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.228694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.228674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-rc6hj" Jul 08 17:34:09.374264 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.374235 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-rc6hj"] Jul 08 17:34:09.377513 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:09.377488 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775929bf_b313_4929_80f1_99005afe292e.slice/crio-559db2a332ad3debea2ba7fb7ccc3288221250482b0eab3e45c86a3e91c1cfb6 WatchSource:0}: Error finding container 559db2a332ad3debea2ba7fb7ccc3288221250482b0eab3e45c86a3e91c1cfb6: Status 404 returned error can't find the container with id 559db2a332ad3debea2ba7fb7ccc3288221250482b0eab3e45c86a3e91c1cfb6 Jul 08 17:34:09.429711 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.429690 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmzgc_e5d590a0-e899-485c-901e-088c7bdabc9a/serve-healthcheck-canary/0.log" Jul 08 17:34:09.582840 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.582811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:09.582983 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.582931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:09.583111 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:09.583087 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:34:09.583217 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:09.583163 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:13.583142771 +0000 UTC m=+56.385229272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : secret "insights-runtime-extractor-tls" not found Jul 08 17:34:09.583369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:09.583352 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c6223a6-2690-4fae-8e29-692c0118f36e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:10.033495 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.033465 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-qrdlt_d0c4509a-bf4d-423b-82d1-72d7f666a32f/kube-storage-version-migrator-operator/0.log" Jul 08 17:34:10.141455 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.141423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" event={"ID":"3bf6faff-4e8f-436a-92b9-8fb3411f3a06","Type":"ContainerStarted","Data":"4e7549edf9ee9e9d5e6670344a3dc79c20fe18d5410690edf4d52aee96da65e3"} Jul 08 17:34:10.141627 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.141476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" event={"ID":"3bf6faff-4e8f-436a-92b9-8fb3411f3a06","Type":"ContainerStarted","Data":"b6034ea0f92215aa163e1108c761d0dd4e948b6583ec5987a6936c188835e0b7"} Jul 08 17:34:10.142962 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.142930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-rc6hj" event={"ID":"775929bf-b313-4929-80f1-99005afe292e","Type":"ContainerStarted","Data":"c3aa0935e7656cf83b9b023a345e569709677cd78758a05d5e0bba2317841571"} Jul 08 17:34:10.143063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.142979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-rc6hj" event={"ID":"775929bf-b313-4929-80f1-99005afe292e","Type":"ContainerStarted","Data":"559db2a332ad3debea2ba7fb7ccc3288221250482b0eab3e45c86a3e91c1cfb6"} Jul 08 17:34:10.160835 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.160790 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-8sz8l" podStartSLOduration=2.326608253 podStartE2EDuration="5.16077736s" totalImagesPullingTime="2.834169107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:10.159612122 +0000 UTC m=+52.961698638" watchObservedRunningTime="2026-07-08 17:34:10.16077736 +0000 UTC m=+52.962863877" Jul 08 17:34:10.177076 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.177038 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-rc6hj" podStartSLOduration=2.177024918 podStartE2EDuration="2.177024918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:10.176559098 +0000 UTC m=+52.978645614" watchObservedRunningTime="2026-07-08 17:34:10.177024918 +0000 UTC m=+52.979111440" Jul 08 17:34:10.391958 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.391893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:10.392244 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.392042 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:34:10.392244 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.392059 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54575979d5-q6smd: secret "image-registry-tls" not found Jul 08 17:34:10.392244 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.392108 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls podName:6d8f09a2-a6d6-492e-9882-9868cd7c18d7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.392093424 +0000 UTC m=+69.194179923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls") pod "image-registry-54575979d5-q6smd" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7") : secret "image-registry-tls" not found Jul 08 17:34:10.492819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:34:10.492930 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:34:10.492930 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:34:10.492930 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:10.492930 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.492934 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.492938 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.492960 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.492987 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert podName:019ebb78-1135-4641-a93e-8b1f5e4e4d6b nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.492968814 +0000 UTC m=+69.295055311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-h7whl" (UID: "019ebb78-1135-4641-a93e-8b1f5e4e4d6b") : secret "networking-console-plugin-cert" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493008 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:10.492950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493010 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls podName:7d531774-6047-4988-9880-c8d5044bfd0d nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.492994358 +0000 UTC m=+69.295080860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-gg28g" (UID: "7d531774-6047-4988-9880-c8d5044bfd0d") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493067 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls podName:0856353e-c7bc-4d49-ae1c-f89082b72417 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.493054968 +0000 UTC m=+69.295141469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-k9nm7" (UID: "0856353e-c7bc-4d49-ae1c-f89082b72417") : secret "samples-operator-tls" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493079 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.493073237 +0000 UTC m=+69.295159733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : secret "router-metrics-certs-default" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493077 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:34:10.493108 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493090 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle podName:e853ca21-cbdf-4916-bd5e-6755ea765ac7 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.493085293 +0000 UTC m=+69.295171789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle") pod "router-default-59c88ffcb5-g8nl9" (UID: "e853ca21-cbdf-4916-bd5e-6755ea765ac7") : configmap references non-existent config key: service-ca.crt Jul 08 17:34:10.493419 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:10.493132 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls podName:b3651a18-f091-4b86-8653-8b1f07453339 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:26.493119317 +0000 UTC m=+69.295205813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls") pod "dns-default-td4sp" (UID: "b3651a18-f091-4b86-8653-8b1f07453339") : secret "dns-default-metrics-tls" not found Jul 08 17:34:13.620735 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:13.620700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:13.621265 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:13.620877 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:34:13.621265 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:13.620954 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls podName:3c6223a6-2690-4fae-8e29-692c0118f36e nodeName:}" failed. No retries permitted until 2026-07-08 17:34:21.620935105 +0000 UTC m=+64.423021623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tn2xt" (UID: "3c6223a6-2690-4fae-8e29-692c0118f36e") : secret "insights-runtime-extractor-tls" not found Jul 08 17:34:14.861066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:14.861035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:34:14.861458 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:14.861444 2579 scope.go:117] "RemoveContainer" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" Jul 08 17:34:14.861655 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:14.861633 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wspx5_openshift-console-operator(20bcd1ae-4584-4f6a-a8ec-90982f45475d)\"" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podUID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" Jul 08 17:34:15.103473 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:15.103436 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:34:15.160486 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:15.160426 2579 scope.go:117] "RemoveContainer" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" Jul 08 17:34:15.160584 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:15.160562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wspx5_openshift-console-operator(20bcd1ae-4584-4f6a-a8ec-90982f45475d)\"" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" podUID="20bcd1ae-4584-4f6a-a8ec-90982f45475d" Jul 08 17:34:19.026306 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:19.026279 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nlrfb" Jul 08 17:34:21.691263 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:21.691223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:21.693715 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:21.693692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c6223a6-2690-4fae-8e29-692c0118f36e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tn2xt\" (UID: \"3c6223a6-2690-4fae-8e29-692c0118f36e\") " pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:21.773077 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:21.773052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9pdx\"" Jul 08 17:34:21.781008 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:21.780992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tn2xt" Jul 08 17:34:21.908483 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:21.908435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tn2xt"] Jul 08 17:34:21.910491 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:21.910465 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6223a6_2690_4fae_8e29_692c0118f36e.slice/crio-613f01c92e784f4bcdac9a3562b2014bf8f305095631ae1e0c21fa822150bea9 WatchSource:0}: Error finding container 613f01c92e784f4bcdac9a3562b2014bf8f305095631ae1e0c21fa822150bea9: Status 404 returned error can't find the container with id 613f01c92e784f4bcdac9a3562b2014bf8f305095631ae1e0c21fa822150bea9 Jul 08 17:34:22.182711 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.182632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tn2xt" event={"ID":"3c6223a6-2690-4fae-8e29-692c0118f36e","Type":"ContainerStarted","Data":"613f01c92e784f4bcdac9a3562b2014bf8f305095631ae1e0c21fa822150bea9"} Jul 08 17:34:22.498337 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.498300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:34:22.498524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.498364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:34:22.501266 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.501072 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:34:22.501266 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.501207 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:34:22.512425 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.512402 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a043eb4-3ebd-42f9-b1b9-0d96dd93553a-original-pull-secret\") pod \"global-pull-secret-syncer-7x7wz\" (UID: \"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a\") " pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:34:22.512564 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.512490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1bc455f-6a8a-4c83-aef5-022768f814f0-metrics-certs\") pod \"network-metrics-daemon-nmqqm\" (UID: \"d1bc455f-6a8a-4c83-aef5-022768f814f0\") " pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:34:22.515371 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.515343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7x7wz" Jul 08 17:34:22.599840 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.599444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:34:22.603583 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.603555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cn2\" (UniqueName: \"kubernetes.io/projected/e3d6ae7b-771d-485a-994d-9d5942fdd7e8-kube-api-access-64cn2\") pod \"network-check-target-75zck\" (UID: \"e3d6ae7b-771d-485a-994d-9d5942fdd7e8\") " pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:34:22.682118 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.682086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7x7wz"] Jul 08 17:34:22.688179 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:22.688150 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a043eb4_3ebd_42f9_b1b9_0d96dd93553a.slice/crio-c80309227c2f8c0e2523d47434330a3a487ff381286ac4c98d2a0b89dab8f90e WatchSource:0}: Error finding container c80309227c2f8c0e2523d47434330a3a487ff381286ac4c98d2a0b89dab8f90e: Status 404 returned error can't find the container with id c80309227c2f8c0e2523d47434330a3a487ff381286ac4c98d2a0b89dab8f90e Jul 08 17:34:22.802414 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.802341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jhm\"" Jul 08 17:34:22.810512 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.810289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmqqm" Jul 08 17:34:22.811423 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.811398 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktdt9\"" Jul 08 17:34:22.819762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.819508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:34:22.988968 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:22.988883 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-75zck"] Jul 08 17:34:22.992025 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:22.991980 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d6ae7b_771d_485a_994d_9d5942fdd7e8.slice/crio-4a836b5ef5758e51d2ec6726f7949cb5cb643538008da1c55dbdb26b03d30085 WatchSource:0}: Error finding container 4a836b5ef5758e51d2ec6726f7949cb5cb643538008da1c55dbdb26b03d30085: Status 404 returned error can't find the container with id 4a836b5ef5758e51d2ec6726f7949cb5cb643538008da1c55dbdb26b03d30085 Jul 08 17:34:23.017854 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.017779 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmqqm"] Jul 08 17:34:23.026298 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:23.025932 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bc455f_6a8a_4c83_aef5_022768f814f0.slice/crio-6affca16143cd8d939b2fccc0dad51120b48fde37243eff2a3b7fbe0d29e2d64 WatchSource:0}: Error finding container 6affca16143cd8d939b2fccc0dad51120b48fde37243eff2a3b7fbe0d29e2d64: Status 404 returned error can't find the container with id 6affca16143cd8d939b2fccc0dad51120b48fde37243eff2a3b7fbe0d29e2d64 Jul 08 17:34:23.187691 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.187637 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmqqm" event={"ID":"d1bc455f-6a8a-4c83-aef5-022768f814f0","Type":"ContainerStarted","Data":"6affca16143cd8d939b2fccc0dad51120b48fde37243eff2a3b7fbe0d29e2d64"} Jul 08 17:34:23.189361 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.189330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75zck" event={"ID":"e3d6ae7b-771d-485a-994d-9d5942fdd7e8","Type":"ContainerStarted","Data":"572d41ad0306c1794e130727563250e0e3949e006097a9c3bc36012dbbf9c921"} Jul 08 17:34:23.189499 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.189381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-75zck" event={"ID":"e3d6ae7b-771d-485a-994d-9d5942fdd7e8","Type":"ContainerStarted","Data":"4a836b5ef5758e51d2ec6726f7949cb5cb643538008da1c55dbdb26b03d30085"} Jul 08 17:34:23.189499 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.189465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:34:23.190710 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.190691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7x7wz" event={"ID":"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a","Type":"ContainerStarted","Data":"c80309227c2f8c0e2523d47434330a3a487ff381286ac4c98d2a0b89dab8f90e"} Jul 08 17:34:23.206315 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:23.206267 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-75zck" podStartSLOduration=65.206249316 podStartE2EDuration="1m5.206249316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:23.205436605 +0000 UTC m=+66.007523123" watchObservedRunningTime="2026-07-08 17:34:23.206249316 +0000 UTC m=+66.008335835" Jul 08 17:34:25.786102 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:25.786074 2579 scope.go:117] "RemoveContainer" containerID="0ef3181a279ec66741afc7eb6542f437b9964a25ab92aa7cfcca8433827f66c1" Jul 08 17:34:26.205320 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.205295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:34:26.205491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.205351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" event={"ID":"20bcd1ae-4584-4f6a-a8ec-90982f45475d","Type":"ContainerStarted","Data":"b0f633d2cb59f36d4a7d67beccb584306af865386bde3265074ed18af9f94641"} Jul 08 17:34:26.205879 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.205820 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:34:26.440998 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.440967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:26.444135 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.444109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"image-registry-54575979d5-q6smd\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:26.542231 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:34:26.542339 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:34:26.542434 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542413 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:34:26.542495 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:26.542551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:26.542551 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.542527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:26.543280 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.543206 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e853ca21-cbdf-4916-bd5e-6755ea765ac7-service-ca-bundle\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:26.545244 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.545222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e853ca21-cbdf-4916-bd5e-6755ea765ac7-metrics-certs\") pod \"router-default-59c88ffcb5-g8nl9\" (UID: \"e853ca21-cbdf-4916-bd5e-6755ea765ac7\") " pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:26.545244 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.545231 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0856353e-c7bc-4d49-ae1c-f89082b72417-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-k9nm7\" (UID: \"0856353e-c7bc-4d49-ae1c-f89082b72417\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:34:26.546278 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.546234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d531774-6047-4988-9880-c8d5044bfd0d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-gg28g\" (UID: \"7d531774-6047-4988-9880-c8d5044bfd0d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:34:26.546714 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.546674 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3651a18-f091-4b86-8653-8b1f07453339-metrics-tls\") pod \"dns-default-td4sp\" (UID: \"b3651a18-f091-4b86-8653-8b1f07453339\") " pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:26.547533 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.547514 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/019ebb78-1135-4641-a93e-8b1f5e4e4d6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-h7whl\" (UID: \"019ebb78-1135-4641-a93e-8b1f5e4e4d6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:34:26.648314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.648238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n999c\"" Jul 08 17:34:26.656336 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.656313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:26.679205 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.679182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p74p\"" Jul 08 17:34:26.687713 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.687396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:26.724682 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.724657 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-chmsj\"" Jul 08 17:34:26.728141 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.728115 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zp2d5\"" Jul 08 17:34:26.732056 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.732032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" Jul 08 17:34:26.736290 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.736177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:26.753677 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.753448 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-f4g57\"" Jul 08 17:34:26.762113 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.761582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" Jul 08 17:34:26.765411 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.765193 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-l79fh\"" Jul 08 17:34:26.773137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.772920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" Jul 08 17:34:26.838194 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.834385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:34:26.863919 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:26.863673 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8f09a2_a6d6_492e_9882_9868cd7c18d7.slice/crio-1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892 WatchSource:0}: Error finding container 1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892: Status 404 returned error can't find the container with id 1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892 Jul 08 17:34:26.868148 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.867451 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-wspx5" Jul 08 17:34:26.888688 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.888650 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-td4sp"] Jul 08 17:34:26.989503 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:26.989481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g"] Jul 08 17:34:27.002229 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:27.002203 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d531774_6047_4988_9880_c8d5044bfd0d.slice/crio-f23cfd4734319269c9917fc34dc46ca76c40156756891501a170a5273b4bf6f2 WatchSource:0}: Error finding container f23cfd4734319269c9917fc34dc46ca76c40156756891501a170a5273b4bf6f2: Status 404 returned error can't find the container with id f23cfd4734319269c9917fc34dc46ca76c40156756891501a170a5273b4bf6f2 Jul 08 17:34:27.044701 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.044622 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7"] Jul 08 17:34:27.199432 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.199405 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59c88ffcb5-g8nl9"] Jul 08 17:34:27.202122 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.202092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-h7whl"] Jul 08 17:34:27.203214 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:27.203185 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode853ca21_cbdf_4916_bd5e_6755ea765ac7.slice/crio-393069e9145672bdb59c8b9d9a22bef329ac5c0e0d5e227c702ad4a5f178c63d WatchSource:0}: Error finding container 393069e9145672bdb59c8b9d9a22bef329ac5c0e0d5e227c702ad4a5f178c63d: Status 404 returned error can't find the container with id 393069e9145672bdb59c8b9d9a22bef329ac5c0e0d5e227c702ad4a5f178c63d Jul 08 17:34:27.206175 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:27.206129 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019ebb78_1135_4641_a93e_8b1f5e4e4d6b.slice/crio-3533f28b43dc9ceb4f56fa5d0fa5cc39ab7510e59c3e1821bc764712ca4dc110 WatchSource:0}: Error finding container 3533f28b43dc9ceb4f56fa5d0fa5cc39ab7510e59c3e1821bc764712ca4dc110: Status 404 returned error can't find the container with id 3533f28b43dc9ceb4f56fa5d0fa5cc39ab7510e59c3e1821bc764712ca4dc110 Jul 08 17:34:27.213126 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.213095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" event={"ID":"0856353e-c7bc-4d49-ae1c-f89082b72417","Type":"ContainerStarted","Data":"2b2d7588e683edad8b793f37b5c2a8a3049ad0c69d5d880143cff6a6d51ea3b9"} Jul 08 17:34:27.215331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.215298 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmqqm" event={"ID":"d1bc455f-6a8a-4c83-aef5-022768f814f0","Type":"ContainerStarted","Data":"5c39b698058caa8b079c1dd104b9482627bb00b4b23e99f385199d01b6ebf59d"} Jul 08 17:34:27.215395 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.215344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmqqm" event={"ID":"d1bc455f-6a8a-4c83-aef5-022768f814f0","Type":"ContainerStarted","Data":"15bbec6ed7e39eaab16194929ee6236a45edc46f5674bb649c85733119259361"} Jul 08 17:34:27.218457 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.218407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tn2xt" event={"ID":"3c6223a6-2690-4fae-8e29-692c0118f36e","Type":"ContainerStarted","Data":"b3a84f24af35920c9c0312e4736af1392cdfb1993d2983828ce0de469d64fdcb"} Jul 08 17:34:27.221066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.221035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-td4sp" event={"ID":"b3651a18-f091-4b86-8653-8b1f07453339","Type":"ContainerStarted","Data":"06d1174944effc3d5fb46453b652d18e18bf9f456b9925bcb03809c3c9cf1710"} Jul 08 17:34:27.222461 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.222413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" event={"ID":"7d531774-6047-4988-9880-c8d5044bfd0d","Type":"ContainerStarted","Data":"f23cfd4734319269c9917fc34dc46ca76c40156756891501a170a5273b4bf6f2"} Jul 08 17:34:27.224628 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.224368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54575979d5-q6smd" event={"ID":"6d8f09a2-a6d6-492e-9882-9868cd7c18d7","Type":"ContainerStarted","Data":"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d"} Jul 08 17:34:27.224628 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.224413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54575979d5-q6smd" event={"ID":"6d8f09a2-a6d6-492e-9882-9868cd7c18d7","Type":"ContainerStarted","Data":"1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892"} Jul 08 17:34:27.224628 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.224438 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:27.227340 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.227287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" event={"ID":"e853ca21-cbdf-4916-bd5e-6755ea765ac7","Type":"ContainerStarted","Data":"393069e9145672bdb59c8b9d9a22bef329ac5c0e0d5e227c702ad4a5f178c63d"} Jul 08 17:34:27.233800 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.233756 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmqqm" podStartSLOduration=65.946331132 podStartE2EDuration="1m9.233735197s" totalImagesPullingTime="3.287404065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:27.233005358 +0000 UTC m=+70.035091878" watchObservedRunningTime="2026-07-08 17:34:27.233735197 +0000 UTC m=+70.035821715" Jul 08 17:34:27.257102 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:27.257044 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54575979d5-q6smd" podStartSLOduration=68.257023607 podStartE2EDuration="1m8.257023607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:27.255076464 +0000 UTC m=+70.057162982" watchObservedRunningTime="2026-07-08 17:34:27.257023607 +0000 UTC m=+70.059110125" Jul 08 17:34:28.233089 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.233037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tn2xt" event={"ID":"3c6223a6-2690-4fae-8e29-692c0118f36e","Type":"ContainerStarted","Data":"30a144da2c9d06b87e30115282902773be79290de05bd54a9a7d02a277e20614"} Jul 08 17:34:28.234629 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.234574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" event={"ID":"e853ca21-cbdf-4916-bd5e-6755ea765ac7","Type":"ContainerStarted","Data":"d7ecde10cc46a3533f1e1eb671a1c7acf8a44a7f04bcd459e1e4222ccc6422c1"} Jul 08 17:34:28.235846 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.235819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" event={"ID":"019ebb78-1135-4641-a93e-8b1f5e4e4d6b","Type":"ContainerStarted","Data":"3533f28b43dc9ceb4f56fa5d0fa5cc39ab7510e59c3e1821bc764712ca4dc110"} Jul 08 17:34:28.260619 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.260541 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podStartSLOduration=58.260503462 podStartE2EDuration="58.260503462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:28.258765149 +0000 UTC m=+71.060851666" watchObservedRunningTime="2026-07-08 17:34:28.260503462 +0000 UTC m=+71.062589982" Jul 08 17:34:28.736487 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.736458 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:28.739211 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.739082 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:28.739211 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:28.739211 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:28.739211 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:28.739211 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:28.739133 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:29.242556 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:29.242278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7x7wz" event={"ID":"4a043eb4-3ebd-42f9-b1b9-0d96dd93553a","Type":"ContainerStarted","Data":"6637210106b4f4baf25b0f0e17af85bc9be68a0a906d29e8170ee7965685bc04"} Jul 08 17:34:29.268494 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:29.268423 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7x7wz" podStartSLOduration=65.220644195 podStartE2EDuration="1m11.26839942s" totalImagesPullingTime="6.047755225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:29.267352722 +0000 UTC m=+72.069439239" watchObservedRunningTime="2026-07-08 17:34:29.26839942 +0000 UTC m=+72.070485941" Jul 08 17:34:29.738439 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:29.738077 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:29.738439 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:29.738439 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:29.738439 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:29.738439 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:29.738136 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:30.738322 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.738280 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:30.738322 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:30.738322 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:30.738322 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:30.738881 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.738350 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:30.952109 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.952062 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj"] Jul 08 17:34:30.995674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.995227 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s"] Jul 08 17:34:30.995674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.995327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.999556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.999818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.999879 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.999879 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:30.999908 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.000273 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 08 17:34:31.002125 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.000288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 08 17:34:31.012571 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.012546 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd"] Jul 08 17:34:31.013726 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.013704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.017129 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.017107 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 17:34:31.017261 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.017109 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4tfrt\"" Jul 08 17:34:31.028725 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.028635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s"] Jul 08 17:34:31.028725 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.028672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd"] Jul 08 17:34:31.028725 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.028691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj"] Jul 08 17:34:31.028725 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.028723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.031638 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.031313 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 17:34:31.043228 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.043186 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-qhmvk"] Jul 08 17:34:31.060915 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.060886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:31.065169 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.065147 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 17:34:31.066015 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.065996 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 17:34:31.073972 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.073945 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gvxx5\"" Jul 08 17:34:31.092278 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.092249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-qhmvk"] Jul 08 17:34:31.093686 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093662 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.093833 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f564d8-176b-49b3-a857-a63e7e6f3b99-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.093833 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.093833 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.094003 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.094003 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.093873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzllq\" (UniqueName: \"kubernetes.io/projected/b9f564d8-176b-49b3-a857-a63e7e6f3b99-kube-api-access-vzllq\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.194294 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.194456 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cec356a-e2d7-4eef-b476-452aa81509ad-tmp\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.194456 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9cec356a-e2d7-4eef-b476-452aa81509ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.194810 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rmw\" (UniqueName: \"kubernetes.io/projected/e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34-kube-api-access-f5rmw\") pod \"downloads-685bfb5f64-qhmvk\" (UID: \"e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34\") " pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:31.194917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.194966 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.194910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzllq\" (UniqueName: \"kubernetes.io/projected/b9f564d8-176b-49b3-a857-a63e7e6f3b99-kube-api-access-vzllq\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.195051 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.195116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8953a6d3-933e-4a99-a269-3db88f882c3d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.195179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9hf\" (UniqueName: \"kubernetes.io/projected/9cec356a-e2d7-4eef-b476-452aa81509ad-kube-api-access-8q9hf\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.195230 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f564d8-176b-49b3-a857-a63e7e6f3b99-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.195230 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.195324 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.195270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtjp\" (UniqueName: \"kubernetes.io/projected/8953a6d3-933e-4a99-a269-3db88f882c3d-kube-api-access-zrtjp\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.198756 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.196772 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f564d8-176b-49b3-a857-a63e7e6f3b99-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.198756 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.197719 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-ca\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.198756 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.198303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.201677 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.201651 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.203209 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.203188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b9f564d8-176b-49b3-a857-a63e7e6f3b99-hub\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.206403 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.206379 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzllq\" (UniqueName: \"kubernetes.io/projected/b9f564d8-176b-49b3-a857-a63e7e6f3b99-kube-api-access-vzllq\") pod \"cluster-proxy-proxy-agent-7dfd9f5498-vwcfj\" (UID: \"b9f564d8-176b-49b3-a857-a63e7e6f3b99\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.296162 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8953a6d3-933e-4a99-a269-3db88f882c3d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.296162 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9hf\" (UniqueName: \"kubernetes.io/projected/9cec356a-e2d7-4eef-b476-452aa81509ad-kube-api-access-8q9hf\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.296342 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtjp\" (UniqueName: \"kubernetes.io/projected/8953a6d3-933e-4a99-a269-3db88f882c3d-kube-api-access-zrtjp\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.296342 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cec356a-e2d7-4eef-b476-452aa81509ad-tmp\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.296342 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9cec356a-e2d7-4eef-b476-452aa81509ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.296481 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rmw\" (UniqueName: \"kubernetes.io/projected/e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34-kube-api-access-f5rmw\") pod \"downloads-685bfb5f64-qhmvk\" (UID: \"e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34\") " pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:31.296699 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.296676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cec356a-e2d7-4eef-b476-452aa81509ad-tmp\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.298950 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.298930 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8953a6d3-933e-4a99-a269-3db88f882c3d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.299093 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.299073 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9cec356a-e2d7-4eef-b476-452aa81509ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.309295 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.309273 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9hf\" (UniqueName: \"kubernetes.io/projected/9cec356a-e2d7-4eef-b476-452aa81509ad-kube-api-access-8q9hf\") pod \"klusterlet-addon-workmgr-7fdc7988f-x2rnd\" (UID: \"9cec356a-e2d7-4eef-b476-452aa81509ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.310387 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.310369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rmw\" (UniqueName: \"kubernetes.io/projected/e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34-kube-api-access-f5rmw\") pod \"downloads-685bfb5f64-qhmvk\" (UID: \"e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34\") " pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:31.310521 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.310503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtjp\" (UniqueName: \"kubernetes.io/projected/8953a6d3-933e-4a99-a269-3db88f882c3d-kube-api-access-zrtjp\") pod \"managed-serviceaccount-addon-agent-5fcbf49456-qhx4s\" (UID: \"8953a6d3-933e-4a99-a269-3db88f882c3d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.317381 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.317359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" Jul 08 17:34:31.328579 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.328559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" Jul 08 17:34:31.346401 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.346373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:31.371672 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.371642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:31.737540 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.737507 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:31.737540 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:31.737540 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:31.737540 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:31.737852 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:31.737576 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:32.738169 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:32.738102 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:32.738169 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:32.738169 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:32.738169 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:32.738169 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:32.738151 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:33.737989 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:33.737946 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:33.737989 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:33.737989 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:33.737989 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:33.738547 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:33.738002 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:34.021518 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.021487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd"] Jul 08 17:34:34.022193 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:34.022010 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cec356a_e2d7_4eef_b476_452aa81509ad.slice/crio-6c18a1c95bfce2f2c91ac1590aea8a6329421083915f2320079b904c20ee6811 WatchSource:0}: Error finding container 6c18a1c95bfce2f2c91ac1590aea8a6329421083915f2320079b904c20ee6811: Status 404 returned error can't find the container with id 6c18a1c95bfce2f2c91ac1590aea8a6329421083915f2320079b904c20ee6811 Jul 08 17:34:34.260751 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.260717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj"] Jul 08 17:34:34.266000 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.265213 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s"] Jul 08 17:34:34.266537 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:34.266326 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9f564d8_176b_49b3_a857_a63e7e6f3b99.slice/crio-1fc784fb81b1c9a20c1a148414442f228519a05509e880ce59a1cc176de90d1f WatchSource:0}: Error finding container 1fc784fb81b1c9a20c1a148414442f228519a05509e880ce59a1cc176de90d1f: Status 404 returned error can't find the container with id 1fc784fb81b1c9a20c1a148414442f228519a05509e880ce59a1cc176de90d1f Jul 08 17:34:34.266671 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.266620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-qhmvk"] Jul 08 17:34:34.268073 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.266972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" event={"ID":"0856353e-c7bc-4d49-ae1c-f89082b72417","Type":"ContainerStarted","Data":"9a09e8f45374e55622d2a0351c170cbf2395692a6ed876b396cba7b90ccafffd"} Jul 08 17:34:34.268073 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.267023 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" event={"ID":"0856353e-c7bc-4d49-ae1c-f89082b72417","Type":"ContainerStarted","Data":"28479a50f81665296c33c7620b06156802f9d7085877fdcb648c3541943745bd"} Jul 08 17:34:34.268073 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:34.268029 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8953a6d3_933e_4a99_a269_3db88f882c3d.slice/crio-984e9398b3b883ad95f9af6ec9eb86d3827cd047e6a90fb0e2b62aff978c7b32 WatchSource:0}: Error finding container 984e9398b3b883ad95f9af6ec9eb86d3827cd047e6a90fb0e2b62aff978c7b32: Status 404 returned error can't find the container with id 984e9398b3b883ad95f9af6ec9eb86d3827cd047e6a90fb0e2b62aff978c7b32 Jul 08 17:34:34.269812 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:34.269765 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9db5d99_bf1f_4a6e_bfe7_e55ed3a41d34.slice/crio-681b89c375536dc749e8040b5e360c71e3ca77d0604d3288118a57d777ce4f99 WatchSource:0}: Error finding container 681b89c375536dc749e8040b5e360c71e3ca77d0604d3288118a57d777ce4f99: Status 404 returned error can't find the container with id 681b89c375536dc749e8040b5e360c71e3ca77d0604d3288118a57d777ce4f99 Jul 08 17:34:34.272758 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.272585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tn2xt" event={"ID":"3c6223a6-2690-4fae-8e29-692c0118f36e","Type":"ContainerStarted","Data":"ed8bc8acdf900c0ea16e9ce999fb1b4e9978d7ce1ef1f140d7d3adb23f261629"} Jul 08 17:34:34.276762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.276314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-td4sp" event={"ID":"b3651a18-f091-4b86-8653-8b1f07453339","Type":"ContainerStarted","Data":"73951b65326a73fd376e4b830dcbe19e3fcbf8324b539812e2cebbb2f24b8397"} Jul 08 17:34:34.276762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.276359 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-td4sp" event={"ID":"b3651a18-f091-4b86-8653-8b1f07453339","Type":"ContainerStarted","Data":"08d84596692c289a3f94e9b8830c45e4b291e9ae0caa3526a2dfd5bac2d24dd9"} Jul 08 17:34:34.276762 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.276397 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:34.278590 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.278550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" event={"ID":"7d531774-6047-4988-9880-c8d5044bfd0d","Type":"ContainerStarted","Data":"c6879217ba9ec728d3de0c428ea218fbd48ade4c482fec1d9a07a0dc7e8d8347"} Jul 08 17:34:34.280544 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.280486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" event={"ID":"9cec356a-e2d7-4eef-b476-452aa81509ad","Type":"ContainerStarted","Data":"6c18a1c95bfce2f2c91ac1590aea8a6329421083915f2320079b904c20ee6811"} Jul 08 17:34:34.281990 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.281958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" event={"ID":"019ebb78-1135-4641-a93e-8b1f5e4e4d6b","Type":"ContainerStarted","Data":"7b0f08368d0d5fdc556da8bd918d1eaa6635b0d34c80378684c5da5749d172df"} Jul 08 17:34:34.290943 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.290899 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-k9nm7" podStartSLOduration=57.673353847 podStartE2EDuration="1m4.290886212s" totalImagesPullingTime="6.617532365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:34.289938678 +0000 UTC m=+77.092025197" watchObservedRunningTime="2026-07-08 17:34:34.290886212 +0000 UTC m=+77.092972708" Jul 08 17:34:34.316386 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.316344 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-gg28g" podStartSLOduration=57.511076606 podStartE2EDuration="1m4.316328668s" totalImagesPullingTime="6.805252062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:34.315754395 +0000 UTC m=+77.117840914" watchObservedRunningTime="2026-07-08 17:34:34.316328668 +0000 UTC m=+77.118415188" Jul 08 17:34:34.340620 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.340128 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-td4sp" podStartSLOduration=33.458151699 podStartE2EDuration="40.340112964s" totalImagesPullingTime="6.881961265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:34.339968796 +0000 UTC m=+77.142055315" watchObservedRunningTime="2026-07-08 17:34:34.340112964 +0000 UTC m=+77.142199482" Jul 08 17:34:34.361497 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.361447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tn2xt" podStartSLOduration=17.642028127 podStartE2EDuration="29.361435847s" totalImagesPullingTime="11.71940772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:34:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:34.360652759 +0000 UTC m=+77.162739277" watchObservedRunningTime="2026-07-08 17:34:34.361435847 +0000 UTC m=+77.163522364" Jul 08 17:34:34.378121 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.378083 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-h7whl" podStartSLOduration=39.785915158 podStartE2EDuration="46.37807278s" totalImagesPullingTime="6.592157622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:34.377799476 +0000 UTC m=+77.179885992" watchObservedRunningTime="2026-07-08 17:34:34.37807278 +0000 UTC m=+77.180159297" Jul 08 17:34:34.536090 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.536060 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl"] Jul 08 17:34:34.563925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.563892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl"] Jul 08 17:34:34.564063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.564017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:34.566953 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.566912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-t5bxn\"" Jul 08 17:34:34.567087 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.566914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 17:34:34.635306 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.635233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48f06d89-7333-407d-b4db-d5e5d9355cf6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-47jdl\" (UID: \"48f06d89-7333-407d-b4db-d5e5d9355cf6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:34.736308 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.736277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48f06d89-7333-407d-b4db-d5e5d9355cf6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-47jdl\" (UID: \"48f06d89-7333-407d-b4db-d5e5d9355cf6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:34.737487 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.737460 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:34.737487 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:34.737487 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:34.737487 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:34.737704 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.737531 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:34.738973 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.738953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/48f06d89-7333-407d-b4db-d5e5d9355cf6-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-47jdl\" (UID: \"48f06d89-7333-407d-b4db-d5e5d9355cf6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:34.873902 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:34.873870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:35.062659 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.062564 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl"] Jul 08 17:34:35.075861 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:35.071463 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f06d89_7333_407d_b4db_d5e5d9355cf6.slice/crio-47c2cfb0c87388544adbee2aa39279aea004b149b1fe2a72e9834878b836d355 WatchSource:0}: Error finding container 47c2cfb0c87388544adbee2aa39279aea004b149b1fe2a72e9834878b836d355: Status 404 returned error can't find the container with id 47c2cfb0c87388544adbee2aa39279aea004b149b1fe2a72e9834878b836d355 Jul 08 17:34:35.294852 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.294810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" event={"ID":"8953a6d3-933e-4a99-a269-3db88f882c3d","Type":"ContainerStarted","Data":"984e9398b3b883ad95f9af6ec9eb86d3827cd047e6a90fb0e2b62aff978c7b32"} Jul 08 17:34:35.296947 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.296897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" event={"ID":"b9f564d8-176b-49b3-a857-a63e7e6f3b99","Type":"ContainerStarted","Data":"1fc784fb81b1c9a20c1a148414442f228519a05509e880ce59a1cc176de90d1f"} Jul 08 17:34:35.298449 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.298421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" event={"ID":"48f06d89-7333-407d-b4db-d5e5d9355cf6","Type":"ContainerStarted","Data":"47c2cfb0c87388544adbee2aa39279aea004b149b1fe2a72e9834878b836d355"} Jul 08 17:34:35.301739 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.300787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-qhmvk" event={"ID":"e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34","Type":"ContainerStarted","Data":"681b89c375536dc749e8040b5e360c71e3ca77d0604d3288118a57d777ce4f99"} Jul 08 17:34:35.738991 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.738591 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:35.738991 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:35.738991 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:35.738991 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:35.738991 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:35.738673 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:36.737249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:36.737208 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:34:36.739468 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:36.739128 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:36.739468 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:36.739468 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:36.739468 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:36.739468 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:36.739177 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:37.738744 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:37.738374 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:37.738744 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:37.738744 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:37.738744 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:37.738744 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:37.738430 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:38.737904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:38.737466 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:38.737904 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:38.737904 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:38.737904 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:38.737904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:38.737534 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:39.737668 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:39.737628 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:39.737668 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:39.737668 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:39.737668 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:39.738244 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:39.737699 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:40.738237 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:40.738200 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:40.738237 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:40.738237 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:40.738237 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:40.738784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:40.738264 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:41.738037 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:41.737818 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:41.738037 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:41.738037 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:41.738037 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:41.738037 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:41.737867 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:42.341084 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.341050 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" event={"ID":"48f06d89-7333-407d-b4db-d5e5d9355cf6","Type":"ContainerStarted","Data":"de0c2b30821c4ed71978f1f50f343e61f30fb881d9811e526f164d35060210b2"} Jul 08 17:34:42.341497 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.341233 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:42.343005 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.342973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" event={"ID":"9cec356a-e2d7-4eef-b476-452aa81509ad","Type":"ContainerStarted","Data":"cd9bf4f11240e8295963be37bcc909f6cbfd4bcf5aec3ece8751609b82f8fc7f"} Jul 08 17:34:42.343150 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.343030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:42.344674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.344648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" event={"ID":"8953a6d3-933e-4a99-a269-3db88f882c3d","Type":"ContainerStarted","Data":"e3e57be05929a2ac0ef80130ef9f271b8fde29428f7fd53a09d047673a9424f6"} Jul 08 17:34:42.346917 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.346893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" event={"ID":"b9f564d8-176b-49b3-a857-a63e7e6f3b99","Type":"ContainerStarted","Data":"7c7cc9d7633fc9c15b43a85f90a5cf99874272414bfd4935584261ad0fb73a69"} Jul 08 17:34:42.347048 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.346932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" event={"ID":"b9f564d8-176b-49b3-a857-a63e7e6f3b99","Type":"ContainerStarted","Data":"b63d02792137ad57514390fa3722046c7b4b06f832c5202db01c094adff379e5"} Jul 08 17:34:42.347048 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.346950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" event={"ID":"b9f564d8-176b-49b3-a857-a63e7e6f3b99","Type":"ContainerStarted","Data":"c520f6031bfbe660eb433168a4a6489e79201468ed45e2e16fe52933d9a914b4"} Jul 08 17:34:42.347048 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.346994 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" Jul 08 17:34:42.347283 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.347264 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" Jul 08 17:34:42.356476 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.356431 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-47jdl" podStartSLOduration=1.803631872 podStartE2EDuration="8.356416564s" totalImagesPullingTime="6.552784692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:42.355637267 +0000 UTC m=+85.157723785" watchObservedRunningTime="2026-07-08 17:34:42.356416564 +0000 UTC m=+85.158503083" Jul 08 17:34:42.371793 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.371747 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fdc7988f-x2rnd" podStartSLOduration=4.748691655 podStartE2EDuration="12.371729977s" totalImagesPullingTime="7.623038322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:42.371329576 +0000 UTC m=+85.173416096" watchObservedRunningTime="2026-07-08 17:34:42.371729977 +0000 UTC m=+85.173816496" Jul 08 17:34:42.404400 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.404343 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fcbf49456-qhx4s" podStartSLOduration=5.042422639 podStartE2EDuration="12.404325838s" totalImagesPullingTime="7.361903199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:42.401336293 +0000 UTC m=+85.203422811" watchObservedRunningTime="2026-07-08 17:34:42.404325838 +0000 UTC m=+85.206412359" Jul 08 17:34:42.425304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.425251 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfd9f5498-vwcfj" podStartSLOduration=5.046191702 podStartE2EDuration="12.425235449s" totalImagesPullingTime="7.379043747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:42.425124388 +0000 UTC m=+85.227210906" watchObservedRunningTime="2026-07-08 17:34:42.425235449 +0000 UTC m=+85.227321969" Jul 08 17:34:42.737795 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.737748 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:42.737795 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:42.737795 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:42.737795 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:42.738080 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:42.737821 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:43.738302 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:43.738262 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:43.738302 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:43.738302 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:43.738302 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:43.738891 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:43.738331 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:44.303985 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:44.303953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-td4sp" Jul 08 17:34:44.738235 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:44.738199 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:44.738235 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:44.738235 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:44.738235 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:44.738814 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:44.738271 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:45.739413 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:45.739374 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:45.739413 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:45.739413 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:45.739413 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:45.739860 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:45.739449 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:46.661980 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:46.661934 2579 patch_prober.go:28] interesting pod/image-registry-54575979d5-q6smd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 17:34:46.662166 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:46.662009 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54575979d5-q6smd" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 17:34:46.738338 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:46.737882 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:46.738338 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:46.738338 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:46.738338 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:46.738338 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:46.737977 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:47.045868 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.045833 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2prw7"] Jul 08 17:34:47.049209 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.049168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.052626 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.052581 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 17:34:47.052822 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.052807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 17:34:47.053211 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.052980 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 17:34:47.053316 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.053301 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-glrk5\"" Jul 08 17:34:47.053556 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.053537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 17:34:47.150299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnkz\" (UniqueName: \"kubernetes.io/projected/26313d58-957b-49df-a547-4fc1df155f5d-kube-api-access-9pnkz\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150537 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150537 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-sys\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150537 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-root\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150537 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-metrics-client-ca\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-wtmp\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.150776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.150676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-textfile\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.251816 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnkz\" (UniqueName: \"kubernetes.io/projected/26313d58-957b-49df-a547-4fc1df155f5d-kube-api-access-9pnkz\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.251816 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-sys\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-root\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.251976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-metrics-client-ca\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-wtmp\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252272 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-textfile\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26313d58-957b-49df-a547-4fc1df155f5d-metrics-client-ca\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252669 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-root\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-sys\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:47.252708 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:34:47.252772 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls podName:26313d58-957b-49df-a547-4fc1df155f5d nodeName:}" failed. No retries permitted until 2026-07-08 17:34:47.752752688 +0000 UTC m=+90.554839184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls") pod "node-exporter-2prw7" (UID: "26313d58-957b-49df-a547-4fc1df155f5d") : secret "node-exporter-tls" not found Jul 08 17:34:47.252819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252785 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-wtmp\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.253208 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.252878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-textfile\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.255370 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.255350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.265649 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.265607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnkz\" (UniqueName: \"kubernetes.io/projected/26313d58-957b-49df-a547-4fc1df155f5d-kube-api-access-9pnkz\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.738138 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.737893 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:47.738138 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:47.738138 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:47.738138 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:47.738138 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.737973 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:47.756314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.756280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.759091 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.759064 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/26313d58-957b-49df-a547-4fc1df155f5d-node-exporter-tls\") pod \"node-exporter-2prw7\" (UID: \"26313d58-957b-49df-a547-4fc1df155f5d\") " pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:47.963445 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:47.963408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2prw7" Jul 08 17:34:48.242366 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:48.242334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:34:48.738405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:48.738365 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:48.738405 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:48.738405 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:48.738405 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:48.738740 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:48.738435 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:49.738334 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:49.738294 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:49.738334 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:49.738334 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:49.738334 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:49.738944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:49.738381 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:50.738277 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:50.738244 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:50.738277 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:50.738277 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:50.738277 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:50.738775 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:50.738305 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:51.737590 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:51.737549 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:51.737590 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:51.737590 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:51.737590 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:51.738015 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:51.737642 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:52.738052 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:52.738011 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:52.738052 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:52.738052 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:52.738052 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:52.738665 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:52.738098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:53.738130 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:53.738088 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:53.738130 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:53.738130 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:53.738130 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:53.739094 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:53.738157 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:54.104401 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:34:54.104368 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26313d58_957b_49df_a547_4fc1df155f5d.slice/crio-def9d0048dd1883aea157d3fb9919ff4adea02baaec79db42953c4732e300e59 WatchSource:0}: Error finding container def9d0048dd1883aea157d3fb9919ff4adea02baaec79db42953c4732e300e59: Status 404 returned error can't find the container with id def9d0048dd1883aea157d3fb9919ff4adea02baaec79db42953c4732e300e59 Jul 08 17:34:54.197121 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:54.197091 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-75zck" Jul 08 17:34:54.415570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:54.415500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2prw7" event={"ID":"26313d58-957b-49df-a547-4fc1df155f5d","Type":"ContainerStarted","Data":"def9d0048dd1883aea157d3fb9919ff4adea02baaec79db42953c4732e300e59"} Jul 08 17:34:54.737585 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:54.737551 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:54.737585 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:54.737585 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:54.737585 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:54.737881 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:54.737633 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:55.422189 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.422147 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-qhmvk" event={"ID":"e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34","Type":"ContainerStarted","Data":"e5d9e496eaafba63942d4c5f0a8c5033b9805bb374eda2226a852ac8d0d2d68e"} Jul 08 17:34:55.422669 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.422313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:55.438584 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.438556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-qhmvk" Jul 08 17:34:55.442372 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.441109 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-qhmvk" podStartSLOduration=4.241817408 podStartE2EDuration="24.441091282s" totalImagesPullingTime="20.199273874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:55.439528298 +0000 UTC m=+98.241614829" watchObservedRunningTime="2026-07-08 17:34:55.441091282 +0000 UTC m=+98.243177800" Jul 08 17:34:55.738414 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.738360 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:55.738414 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:55.738414 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:55.738414 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:55.738742 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:55.738445 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:56.428683 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:56.428648 2579 generic.go:356] "Generic (PLEG): container finished" podID="26313d58-957b-49df-a547-4fc1df155f5d" containerID="1742fd730394c170652c86b48e32b602b8e373ccdf7778bebb6f65290fbee207" exitCode=0 Jul 08 17:34:56.429144 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:56.428695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2prw7" event={"ID":"26313d58-957b-49df-a547-4fc1df155f5d","Type":"ContainerDied","Data":"1742fd730394c170652c86b48e32b602b8e373ccdf7778bebb6f65290fbee207"} Jul 08 17:34:56.738299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:56.738260 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:56.738299 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:56.738299 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:56.738299 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:56.738505 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:56.738323 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:57.436491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:57.436451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2prw7" event={"ID":"26313d58-957b-49df-a547-4fc1df155f5d","Type":"ContainerStarted","Data":"d2552e72506d2c812b00e48d2e33645c04086c900f2c4aaa47698e79c0811969"} Jul 08 17:34:57.436901 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:57.436523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2prw7" event={"ID":"26313d58-957b-49df-a547-4fc1df155f5d","Type":"ContainerStarted","Data":"7abef6c2fdc53830ccb763766962ba6feb3087acf6093e8f642673af334766b6"} Jul 08 17:34:57.458632 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:57.458560 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2prw7" podStartSLOduration=8.959685855 podStartE2EDuration="10.458538572s" totalImagesPullingTime="1.385054206s" totalInitContainerRuntime="113.798511ms" isStatefulPod=true podCreationTimestamp="2026-07-08 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:57.456811649 +0000 UTC m=+100.258898192" watchObservedRunningTime="2026-07-08 17:34:57.458538572 +0000 UTC m=+100.260625090" Jul 08 17:34:57.738417 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:57.738385 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:57.738417 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:57.738417 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:57.738417 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:57.738792 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:57.738439 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:58.438236 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:58.438203 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:34:58.737573 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:58.737542 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:58.737573 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:58.737573 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:58.737573 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:58.737907 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:58.737628 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:34:59.737832 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:59.737795 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:34:59.737832 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:34:59.737832 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:34:59.737832 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:34:59.738307 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:34:59.737873 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:00.737904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:00.737864 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:00.737904 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:00.737904 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:00.737904 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:00.738430 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:00.737932 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:01.738228 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:01.738191 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:01.738228 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:01.738228 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:01.738228 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:01.738742 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:01.738251 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:02.738040 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:02.738006 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:02.738040 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:02.738040 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:02.738040 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:02.738589 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:02.738071 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:03.738433 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:03.738393 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:03.738433 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:03.738433 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:03.738433 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:03.738889 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:03.738448 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:04.738316 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:04.738284 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:04.738316 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:04.738316 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:04.738316 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:04.738883 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:04.738344 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:05.737783 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:05.737749 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:05.737783 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:05.737783 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:05.737783 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:05.738106 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:05.737809 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:06.738147 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:06.738113 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:06.738147 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:06.738147 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:06.738147 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:06.738563 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:06.738180 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:07.737625 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:07.737579 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:07.737625 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:07.737625 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:07.737625 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:07.737857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:07.737643 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:08.737641 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:08.737589 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:08.737641 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:08.737641 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:08.737641 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:08.738086 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:08.737661 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:09.738253 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:09.738214 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:09.738253 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:09.738253 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:09.738253 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:09.738707 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:09.738275 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:10.738309 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:10.738267 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:10.738309 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:10.738309 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:10.738309 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:10.738952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:10.738341 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:11.737843 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:11.737804 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:11.737843 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:11.737843 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:11.737843 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:11.738222 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:11.737858 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:12.737975 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:12.737944 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:12.737975 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:12.737975 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:12.737975 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:12.738525 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:12.737993 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:13.738259 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:13.738221 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:13.738259 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:13.738259 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:13.738259 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:13.738836 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:13.738286 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:14.737607 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:14.737575 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:14.737607 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:14.737607 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:14.737607 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:14.737861 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:14.737654 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:15.737687 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:15.737644 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:15.737687 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:15.737687 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:15.737687 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:15.738142 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:15.737717 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:16.738239 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:16.738200 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:16.738239 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:16.738239 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:16.738239 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:16.738709 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:16.738261 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:17.737621 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:17.737575 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:17.737621 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:17.737621 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:17.737621 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:17.737944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:17.737653 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:18.738220 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:18.738183 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:18.738220 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:18.738220 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:18.738220 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:18.738850 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:18.738248 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:19.737528 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:19.737488 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:19.737528 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:19.737528 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:19.737528 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:19.737814 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:19.737549 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:20.737411 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:20.737376 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:20.737411 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:20.737411 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:20.737411 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:20.737896 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:20.737433 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:21.737890 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:21.737853 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:21.737890 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:21.737890 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:21.737890 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:21.738341 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:21.737924 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:22.737628 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:22.737578 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:22.737628 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:22.737628 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:22.737628 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:22.737867 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:22.737653 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:23.464156 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.464105 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54575979d5-q6smd" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerName="registry" containerID="cri-o://9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d" gracePeriod=30 Jul 08 17:35:23.737794 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.737766 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:23.737794 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:23.737794 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:23.737794 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:23.738017 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.737820 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:23.752937 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.752907 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:35:23.858586 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858586 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858632 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858747 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z87j\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858773 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.858857 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.858798 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted\") pod \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\" (UID: \"6d8f09a2-a6d6-492e-9882-9868cd7c18d7\") " Jul 08 17:35:23.859207 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.859133 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:35:23.859292 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.859265 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:35:23.861664 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.861632 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:35:23.861795 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.861767 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j" (OuterVolumeSpecName: "kube-api-access-4z87j") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "kube-api-access-4z87j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:35:23.861847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.861785 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:35:23.861847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.861820 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:35:23.861916 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.861855 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:35:23.866902 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.866878 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6d8f09a2-a6d6-492e-9882-9868cd7c18d7" (UID: "6d8f09a2-a6d6-492e-9882-9868cd7c18d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:35:23.959896 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959863 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-trusted-ca\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.959896 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959893 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z87j\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-kube-api-access-4z87j\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.959896 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959904 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-image-registry-private-configuration\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.960116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959912 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-ca-trust-extracted\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.960116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959921 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-installation-pull-secrets\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.960116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959929 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-tls\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.960116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959937 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-registry-certificates\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:23.960116 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:23.959944 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8f09a2-a6d6-492e-9882-9868cd7c18d7-bound-sa-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:35:24.541638 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.541582 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerID="9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d" exitCode=0 Jul 08 17:35:24.542137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.541648 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54575979d5-q6smd" Jul 08 17:35:24.542137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.541670 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54575979d5-q6smd" event={"ID":"6d8f09a2-a6d6-492e-9882-9868cd7c18d7","Type":"ContainerDied","Data":"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d"} Jul 08 17:35:24.542137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.541718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54575979d5-q6smd" event={"ID":"6d8f09a2-a6d6-492e-9882-9868cd7c18d7","Type":"ContainerDied","Data":"1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892"} Jul 08 17:35:24.542137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.541738 2579 scope.go:117] "RemoveContainer" containerID="9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d" Jul 08 17:35:24.555381 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.555356 2579 scope.go:117] "RemoveContainer" containerID="9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d" Jul 08 17:35:24.555718 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:35:24.555695 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d\": container with ID starting with 9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d not found: ID does not exist" containerID="9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d" Jul 08 17:35:24.555809 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.555724 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d"} err="failed to get container status \"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d\": rpc error: code = NotFound desc = could not find container \"9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d\": container with ID starting with 9aa7900eba1da5bbe0efbb91e366067df291ec3e7aeee1e70d7cc65ac6cf431d not found: ID does not exist" Jul 08 17:35:24.568426 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.568402 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:35:24.572289 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.572268 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54575979d5-q6smd"] Jul 08 17:35:24.578333 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:35:24.578308 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8f09a2_a6d6_492e_9882_9868cd7c18d7.slice/crio-1f249bcd878e59c570040a09e606e554c8a8b947610b4c58077f686e7f981892\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8f09a2_a6d6_492e_9882_9868cd7c18d7.slice\": RecentStats: unable to find data in memory cache]" Jul 08 17:35:24.738307 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.738264 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:24.738307 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:24.738307 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:24.738307 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:24.738549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:24.738327 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:25.737725 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:25.737694 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:25.737725 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:25.737725 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:25.737725 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:25.738215 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:25.737746 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:25.790609 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:25.790569 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" path="/var/lib/kubelet/pods/6d8f09a2-a6d6-492e-9882-9868cd7c18d7/volumes" Jul 08 17:35:26.737678 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:26.737644 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:26.737678 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:26.737678 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:26.737678 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:26.738178 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:26.737700 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:27.738089 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:27.738052 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:27.738089 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:27.738089 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:27.738089 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:27.738518 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:27.738111 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:28.737850 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:28.737814 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:28.737850 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:28.737850 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:28.737850 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:28.738124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:28.737870 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:29.737620 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:29.737559 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:29.737620 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:29.737620 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:29.737620 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:29.737942 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:29.737645 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:30.567109 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:30.567076 2579 generic.go:356] "Generic (PLEG): container finished" podID="d0c4509a-bf4d-423b-82d1-72d7f666a32f" containerID="91baba7baa33ff7f7d23eb7f11fa2fb161c726d8e041becf2e82ae7bbee07d64" exitCode=0 Jul 08 17:35:30.567499 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:30.567124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" event={"ID":"d0c4509a-bf4d-423b-82d1-72d7f666a32f","Type":"ContainerDied","Data":"91baba7baa33ff7f7d23eb7f11fa2fb161c726d8e041becf2e82ae7bbee07d64"} Jul 08 17:35:30.567543 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:30.567523 2579 scope.go:117] "RemoveContainer" containerID="91baba7baa33ff7f7d23eb7f11fa2fb161c726d8e041becf2e82ae7bbee07d64" Jul 08 17:35:30.737905 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:30.737866 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:30.737905 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:30.737905 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:30.737905 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:30.738169 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:30.737934 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:31.573574 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.573535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-qrdlt" event={"ID":"d0c4509a-bf4d-423b-82d1-72d7f666a32f","Type":"ContainerStarted","Data":"35426743cff5fc64a87543c765f6bdb2668c132ba315186374a841c9e5bf68c4"} Jul 08 17:35:31.575010 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.574984 2579 generic.go:356] "Generic (PLEG): container finished" podID="1238bd9b-fa90-4059-89ff-f57876652cde" containerID="0e6eeb09aa5e9a1d43f952c5e378a7c28bd2198fe10387acd728db6451ca5e57" exitCode=0 Jul 08 17:35:31.575130 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.575053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" event={"ID":"1238bd9b-fa90-4059-89ff-f57876652cde","Type":"ContainerDied","Data":"0e6eeb09aa5e9a1d43f952c5e378a7c28bd2198fe10387acd728db6451ca5e57"} Jul 08 17:35:31.575398 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.575382 2579 scope.go:117] "RemoveContainer" containerID="0e6eeb09aa5e9a1d43f952c5e378a7c28bd2198fe10387acd728db6451ca5e57" Jul 08 17:35:31.738345 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.738298 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:31.738345 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:31.738345 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:31.738345 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:31.738650 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:31.738368 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:32.581307 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:32.581275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-l82dg" event={"ID":"1238bd9b-fa90-4059-89ff-f57876652cde","Type":"ContainerStarted","Data":"139a73bc69338c8a41a15ccc53d66882e287d5f7e776780ff2ad3a16a2766c33"} Jul 08 17:35:32.737314 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:32.737280 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:32.737314 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:32.737314 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:32.737314 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:32.737663 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:32.737348 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:33.738078 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:33.738032 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:33.738078 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:33.738078 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:33.738078 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:33.738544 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:33.738111 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:34.737730 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:34.737698 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:34.737730 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:34.737730 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:34.737730 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:34.737993 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:34.737751 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:35.594773 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:35.594685 2579 generic.go:356] "Generic (PLEG): container finished" podID="c32f2368-e81f-42ea-804f-62c5d495c6a8" containerID="51367225fff693904bb251f338e91c9905b6909fdd1263516c3df96c7b556c23" exitCode=0 Jul 08 17:35:35.594773 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:35.594761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" event={"ID":"c32f2368-e81f-42ea-804f-62c5d495c6a8","Type":"ContainerDied","Data":"51367225fff693904bb251f338e91c9905b6909fdd1263516c3df96c7b556c23"} Jul 08 17:35:35.595203 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:35.595121 2579 scope.go:117] "RemoveContainer" containerID="51367225fff693904bb251f338e91c9905b6909fdd1263516c3df96c7b556c23" Jul 08 17:35:35.737764 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:35.737732 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:35.737764 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:35.737764 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:35.737764 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:35.738024 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:35.737787 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:36.317657 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:36.317621 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59c88ffcb5-g8nl9_e853ca21-cbdf-4916-bd5e-6755ea765ac7/router/0.log" Jul 08 17:35:36.333081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:36.333052 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmzgc_e5d590a0-e899-485c-901e-088c7bdabc9a/serve-healthcheck-canary/0.log" Jul 08 17:35:36.600928 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:36.600842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-cjdr8" event={"ID":"c32f2368-e81f-42ea-804f-62c5d495c6a8","Type":"ContainerStarted","Data":"6ff585ba064298dc132371c056d1988d56d0de193644966f8af5793ea1db8d30"} Jul 08 17:35:36.738146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:36.738111 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:36.738146 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:36.738146 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:36.738146 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:36.738385 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:36.738175 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:37.738110 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:37.738073 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:37.738110 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:37.738110 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:37.738110 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:37.738557 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:37.738135 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:38.737941 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:38.737897 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:38.737941 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:38.737941 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:38.737941 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:38.738412 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:38.737959 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:39.738298 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:39.738259 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:39.738298 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:39.738298 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:39.738298 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:39.738829 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:39.738322 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:40.738153 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:40.738117 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:40.738153 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:40.738153 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:40.738153 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:40.738650 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:40.738188 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:41.737783 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:41.737746 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:41.737783 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:41.737783 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:41.737783 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:41.738033 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:41.737807 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:42.738443 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:42.738402 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:42.738443 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:42.738443 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:42.738443 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:42.739020 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:42.738484 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:43.737769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:43.737731 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:43.737769 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:43.737769 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:43.737769 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:43.738051 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:43.737790 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:44.737776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:44.737738 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:44.737776 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:44.737776 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:44.737776 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:44.738291 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:44.737796 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:45.737528 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:45.737485 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:45.737528 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:45.737528 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:45.737528 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:45.737809 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:45.737550 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:46.737533 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:46.737487 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:46.737533 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:46.737533 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:46.737533 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:46.738245 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:46.737572 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:47.738188 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:47.738138 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:47.738188 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:47.738188 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:47.738188 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:47.738924 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:47.738216 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:48.738568 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:48.738523 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:48.738568 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:48.738568 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:48.738568 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:48.739158 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:48.738627 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:49.738028 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:49.737984 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:49.738028 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:49.738028 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:49.738028 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:49.738350 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:49.738068 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:50.737450 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:50.737412 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:50.737450 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:50.737450 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:50.737450 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:50.737946 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:50.737474 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:51.738212 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:51.738175 2579 patch_prober.go:28] interesting pod/router-default-59c88ffcb5-g8nl9 container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 08 17:35:51.738212 ip-10-0-129-65 kubenswrapper[2579]: [-]has-synced failed: reason withheld Jul 08 17:35:51.738212 ip-10-0-129-65 kubenswrapper[2579]: [+]process-running ok Jul 08 17:35:51.738212 ip-10-0-129-65 kubenswrapper[2579]: healthz check failed Jul 08 17:35:51.738706 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:51.738239 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" podUID="e853ca21-cbdf-4916-bd5e-6755ea765ac7" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 17:35:52.738318 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:52.738285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:35:52.739614 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:35:52.739577 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59c88ffcb5-g8nl9" Jul 08 17:38:17.634561 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:38:17.634529 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:38:17.637949 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:38:17.637926 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:38:17.645179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:38:17.645161 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 17:43:12.382769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.382741 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4"] Jul 08 17:43:12.383196 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.383057 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerName="registry" Jul 08 17:43:12.383196 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.383067 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerName="registry" Jul 08 17:43:12.383196 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.383115 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8f09a2-a6d6-492e-9882-9868cd7c18d7" containerName="registry" Jul 08 17:43:12.384894 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.384879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.387861 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.387839 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 17:43:12.388343 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.388324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cfbk9\"" Jul 08 17:43:12.388694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.388673 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 17:43:12.388796 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.388779 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 17:43:12.388852 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.388800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 17:43:12.424911 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.424880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4"] Jul 08 17:43:12.448969 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.448939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.449095 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.448987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.449095 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.449032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.449095 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.449050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.449095 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.449067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9g4\" (UniqueName: \"kubernetes.io/projected/b37139c1-69bb-4dda-87f4-ec27229b72ef-kube-api-access-2b9g4\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549362 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549541 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549541 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549541 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549541 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9g4\" (UniqueName: \"kubernetes.io/projected/b37139c1-69bb-4dda-87f4-ec27229b72ef-kube-api-access-2b9g4\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.549827 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.549800 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.552023 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.551996 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b37139c1-69bb-4dda-87f4-ec27229b72ef-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.552134 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.552110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.552134 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.552126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b37139c1-69bb-4dda-87f4-ec27229b72ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.558333 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.558309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9g4\" (UniqueName: \"kubernetes.io/projected/b37139c1-69bb-4dda-87f4-ec27229b72ef-kube-api-access-2b9g4\") pod \"opendatahub-operator-controller-manager-5c548d5447-99nm4\" (UID: \"b37139c1-69bb-4dda-87f4-ec27229b72ef\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.695159 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.695125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:12.834428 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.834403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4"] Jul 08 17:43:12.837232 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:12.837207 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37139c1_69bb_4dda_87f4_ec27229b72ef.slice/crio-75497575233c393e7893f5e541e903796ea05b6e17f5cedd562453ac2b03977e WatchSource:0}: Error finding container 75497575233c393e7893f5e541e903796ea05b6e17f5cedd562453ac2b03977e: Status 404 returned error can't find the container with id 75497575233c393e7893f5e541e903796ea05b6e17f5cedd562453ac2b03977e Jul 08 17:43:12.839006 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:12.838991 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:43:13.365865 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:13.365829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" event={"ID":"b37139c1-69bb-4dda-87f4-ec27229b72ef","Type":"ContainerStarted","Data":"75497575233c393e7893f5e541e903796ea05b6e17f5cedd562453ac2b03977e"} Jul 08 17:43:16.381669 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:16.381628 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" event={"ID":"b37139c1-69bb-4dda-87f4-ec27229b72ef","Type":"ContainerStarted","Data":"f5481d382bf2430cb5db0b916a2fe4e379eeb5efa4baae79dffc46bdce7ada37"} Jul 08 17:43:17.387204 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:17.387171 2579 generic.go:356] "Generic (PLEG): container finished" podID="b37139c1-69bb-4dda-87f4-ec27229b72ef" containerID="f5481d382bf2430cb5db0b916a2fe4e379eeb5efa4baae79dffc46bdce7ada37" exitCode=0 Jul 08 17:43:17.387591 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:17.387231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" event={"ID":"b37139c1-69bb-4dda-87f4-ec27229b72ef","Type":"ContainerDied","Data":"f5481d382bf2430cb5db0b916a2fe4e379eeb5efa4baae79dffc46bdce7ada37"} Jul 08 17:43:17.667440 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:17.667364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:43:17.667440 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:17.667365 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:43:18.393586 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:18.393549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" event={"ID":"b37139c1-69bb-4dda-87f4-ec27229b72ef","Type":"ContainerStarted","Data":"7a63fd0b5826583fa036189cb862874a7562893294dfc66a566fe9253a23329b"} Jul 08 17:43:18.394033 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:18.393716 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:18.415106 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:18.415052 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" podStartSLOduration=2.076635152 podStartE2EDuration="6.415034873s" totalImagesPullingTime="3.308349572s" totalInitContainerRuntime="1.030050149s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:18.41300655 +0000 UTC m=+601.215093069" watchObservedRunningTime="2026-07-08 17:43:18.415034873 +0000 UTC m=+601.217121391" Jul 08 17:43:29.401278 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:29.401247 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-99nm4" Jul 08 17:43:33.078260 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.078226 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-89v6r"] Jul 08 17:43:33.088229 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.088207 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.091872 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.091849 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 17:43:33.094809 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.094790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w6l9g\"" Jul 08 17:43:33.095549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.095531 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 17:43:33.120769 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.120739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-89v6r"] Jul 08 17:43:33.209668 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.209587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhzq\" (UniqueName: \"kubernetes.io/projected/29f5b358-7821-49d3-98a1-52a170ee8960-kube-api-access-clhzq\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.209668 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.209665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5b358-7821-49d3-98a1-52a170ee8960-tls-certs\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.209905 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.209730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29f5b358-7821-49d3-98a1-52a170ee8960-tmp\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.310729 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.310689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29f5b358-7821-49d3-98a1-52a170ee8960-tmp\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.310729 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.310733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhzq\" (UniqueName: \"kubernetes.io/projected/29f5b358-7821-49d3-98a1-52a170ee8960-kube-api-access-clhzq\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.311008 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.310762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5b358-7821-49d3-98a1-52a170ee8960-tls-certs\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.313321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.313285 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29f5b358-7821-49d3-98a1-52a170ee8960-tmp\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.313451 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.313390 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f5b358-7821-49d3-98a1-52a170ee8960-tls-certs\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.345700 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.345622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhzq\" (UniqueName: \"kubernetes.io/projected/29f5b358-7821-49d3-98a1-52a170ee8960-kube-api-access-clhzq\") pod \"kube-auth-proxy-6674766f74-89v6r\" (UID: \"29f5b358-7821-49d3-98a1-52a170ee8960\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.397298 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.397263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" Jul 08 17:43:33.408186 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.408154 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5txw"] Jul 08 17:43:33.412187 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.412161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:33.415944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.415918 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 17:43:33.416552 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.416535 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dlklh\"" Jul 08 17:43:33.453669 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.453639 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5txw"] Jul 08 17:43:33.513223 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.513191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrtc\" (UniqueName: \"kubernetes.io/projected/686114d0-cc38-45b7-89f5-ee98b6b42723-kube-api-access-svrtc\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:33.513438 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.513291 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:33.562339 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.562301 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-89v6r"] Jul 08 17:43:33.565503 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:33.565475 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f5b358_7821_49d3_98a1_52a170ee8960.slice/crio-ff130e0c9cbe1084c42148e7429ef978f4e1484401b093987a8fd56d583edc20 WatchSource:0}: Error finding container ff130e0c9cbe1084c42148e7429ef978f4e1484401b093987a8fd56d583edc20: Status 404 returned error can't find the container with id ff130e0c9cbe1084c42148e7429ef978f4e1484401b093987a8fd56d583edc20 Jul 08 17:43:33.613929 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.613844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:33.614095 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.613929 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrtc\" (UniqueName: \"kubernetes.io/projected/686114d0-cc38-45b7-89f5-ee98b6b42723-kube-api-access-svrtc\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:33.614095 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:33.613998 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 08 17:43:33.614095 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:33.614072 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert podName:686114d0-cc38-45b7-89f5-ee98b6b42723 nodeName:}" failed. No retries permitted until 2026-07-08 17:43:34.114057089 +0000 UTC m=+616.916143585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert") pod "odh-model-controller-7998c67dc4-r5txw" (UID: "686114d0-cc38-45b7-89f5-ee98b6b42723") : secret "odh-model-controller-webhook-cert" not found Jul 08 17:43:33.638254 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:33.638217 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrtc\" (UniqueName: \"kubernetes.io/projected/686114d0-cc38-45b7-89f5-ee98b6b42723-kube-api-access-svrtc\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:34.118017 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.117964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:34.120636 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.120592 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/686114d0-cc38-45b7-89f5-ee98b6b42723-cert\") pod \"odh-model-controller-7998c67dc4-r5txw\" (UID: \"686114d0-cc38-45b7-89f5-ee98b6b42723\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:34.330491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.330459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:34.412579 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.412359 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x4f7b"] Jul 08 17:43:34.417013 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.416980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.420308 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:34.420271 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"model-serving-api-54d89bd5fc-x4f7b\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="194fd4db-b502-40ce-b463-9ef5d3339207" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.420473 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:34.420298 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"model-serving-api-tls\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" Jul 08 17:43:34.453783 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.453745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x4f7b"] Jul 08 17:43:34.471350 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.471281 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" event={"ID":"29f5b358-7821-49d3-98a1-52a170ee8960","Type":"ContainerStarted","Data":"ff130e0c9cbe1084c42148e7429ef978f4e1484401b093987a8fd56d583edc20"} Jul 08 17:43:34.520778 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.520644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7wv\" (UniqueName: \"kubernetes.io/projected/194fd4db-b502-40ce-b463-9ef5d3339207-kube-api-access-9g7wv\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.520778 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.520706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.529836 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.529809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5txw"] Jul 08 17:43:34.532622 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:34.532551 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686114d0_cc38_45b7_89f5_ee98b6b42723.slice/crio-7085d6a26d1dda1a2629daedf2ce8a166df1e0dd9adec1e2ab43b04d500df294 WatchSource:0}: Error finding container 7085d6a26d1dda1a2629daedf2ce8a166df1e0dd9adec1e2ab43b04d500df294: Status 404 returned error can't find the container with id 7085d6a26d1dda1a2629daedf2ce8a166df1e0dd9adec1e2ab43b04d500df294 Jul 08 17:43:34.622080 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.622044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7wv\" (UniqueName: \"kubernetes.io/projected/194fd4db-b502-40ce-b463-9ef5d3339207-kube-api-access-9g7wv\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.622289 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.622107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:34.635489 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:34.635458 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7wv\" (UniqueName: \"kubernetes.io/projected/194fd4db-b502-40ce-b463-9ef5d3339207-kube-api-access-9g7wv\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:35.046275 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.046237 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:35.050137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.050111 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.052749 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.052717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 17:43:35.052903 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.052828 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 17:43:35.052903 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.052719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 17:43:35.053066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.052974 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 17:43:35.053066 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.052996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-59bzv\"" Jul 08 17:43:35.063305 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.063278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:35.126869 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.126869 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.127303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.127303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.127303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.127303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.127303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.126992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v64w\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228087 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228087 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.228369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.228232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v64w\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.229178 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.229150 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.231128 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.231104 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.231248 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.231159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.231312 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.231244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.231438 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.231417 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.239963 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.239937 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v64w\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.240086 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.240066 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token\") pod \"istiod-openshift-gateway-7574459656-ltfrf\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.363483 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.363394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:35.479834 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.479794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" event={"ID":"686114d0-cc38-45b7-89f5-ee98b6b42723","Type":"ContainerStarted","Data":"7085d6a26d1dda1a2629daedf2ce8a166df1e0dd9adec1e2ab43b04d500df294"} Jul 08 17:43:35.542564 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.542241 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:35.622706 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:35.622621 2579 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: failed to sync secret cache: timed out waiting for the condition Jul 08 17:43:35.622893 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:35.622726 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs podName:194fd4db-b502-40ce-b463-9ef5d3339207 nodeName:}" failed. No retries permitted until 2026-07-08 17:43:36.122701643 +0000 UTC m=+618.924788154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs") pod "model-serving-api-54d89bd5fc-x4f7b" (UID: "194fd4db-b502-40ce-b463-9ef5d3339207") : failed to sync secret cache: timed out waiting for the condition Jul 08 17:43:35.694117 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:35.693876 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 17:43:36.138618 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:36.138571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:36.141462 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:36.141431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/194fd4db-b502-40ce-b463-9ef5d3339207-tls-certs\") pod \"model-serving-api-54d89bd5fc-x4f7b\" (UID: \"194fd4db-b502-40ce-b463-9ef5d3339207\") " pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:36.229711 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:36.229665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:36.485948 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:36.485910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" event={"ID":"14c32628-1f6b-483a-8809-ad3e7facc637","Type":"ContainerStarted","Data":"204ad5478e886cf990799872035233250fe8a786dff35d20d44acc50181f1f3c"} Jul 08 17:43:38.269261 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.269220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-x4f7b"] Jul 08 17:43:38.275267 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:38.275227 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194fd4db_b502_40ce_b463_9ef5d3339207.slice/crio-870054dd086997b633737637511f3989813b92d8abf1f50d0ad9f2dc5d0718d5 WatchSource:0}: Error finding container 870054dd086997b633737637511f3989813b92d8abf1f50d0ad9f2dc5d0718d5: Status 404 returned error can't find the container with id 870054dd086997b633737637511f3989813b92d8abf1f50d0ad9f2dc5d0718d5 Jul 08 17:43:38.497876 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.497837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" event={"ID":"29f5b358-7821-49d3-98a1-52a170ee8960","Type":"ContainerStarted","Data":"753c2d30a638f1bcf6e96f408fc32964c8fcc0c01427137c3788238b60345085"} Jul 08 17:43:38.500690 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.500651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" event={"ID":"686114d0-cc38-45b7-89f5-ee98b6b42723","Type":"ContainerStarted","Data":"2a75cf4027486f0fcbfc756d4ea01cf5ef110fdf4c4a8e9195fb12798fd68dc8"} Jul 08 17:43:38.500867 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.500728 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:38.502619 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.502549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" event={"ID":"194fd4db-b502-40ce-b463-9ef5d3339207","Type":"ContainerStarted","Data":"870054dd086997b633737637511f3989813b92d8abf1f50d0ad9f2dc5d0718d5"} Jul 08 17:43:38.521652 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.521573 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6674766f74-89v6r" podStartSLOduration=2.011408338 podStartE2EDuration="6.521551869s" totalImagesPullingTime="4.510143531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:38.519170616 +0000 UTC m=+621.321257136" watchObservedRunningTime="2026-07-08 17:43:38.521551869 +0000 UTC m=+621.323638389" Jul 08 17:43:38.538236 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:38.538160 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" podStartSLOduration=1.948054171 podStartE2EDuration="5.538139077s" totalImagesPullingTime="3.590084906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:38.535891236 +0000 UTC m=+621.337977758" watchObservedRunningTime="2026-07-08 17:43:38.538139077 +0000 UTC m=+621.340225597" Jul 08 17:43:39.495573 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:39.495533 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 08 17:43:39.495880 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:39.495615 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 08 17:43:40.519479 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:40.519444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" event={"ID":"14c32628-1f6b-483a-8809-ad3e7facc637","Type":"ContainerStarted","Data":"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca"} Jul 08 17:43:40.519915 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:40.519650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:40.521300 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:40.521263 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-ltfrf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 17:43:40.521422 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:40.521333 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 17:43:40.551099 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:40.551044 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" podStartSLOduration=1.60361348 podStartE2EDuration="5.55102748s" totalImagesPullingTime="3.947414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:40.550739971 +0000 UTC m=+623.352826490" watchObservedRunningTime="2026-07-08 17:43:40.55102748 +0000 UTC m=+623.353114004" Jul 08 17:43:41.526306 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:41.526258 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" event={"ID":"194fd4db-b502-40ce-b463-9ef5d3339207","Type":"ContainerStarted","Data":"bdc21b9e64fae1620eb2ccb559a4542326a652eeb281c317e9fb185ca926034b"} Jul 08 17:43:41.526801 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:41.526583 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:43:41.527083 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:41.527059 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-ltfrf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 17:43:41.527137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:41.527118 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 17:43:41.573502 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:41.573447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" podStartSLOduration=4.523944661 podStartE2EDuration="7.573429895s" totalImagesPullingTime="3.049485234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:41.570080359 +0000 UTC m=+624.372166879" watchObservedRunningTime="2026-07-08 17:43:41.573429895 +0000 UTC m=+624.375516413" Jul 08 17:43:43.889439 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.889393 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z"] Jul 08 17:43:43.902510 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.902471 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:43.905424 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.905404 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8lqcq\"" Jul 08 17:43:43.905715 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.905698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 17:43:43.905807 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.905708 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln"] Jul 08 17:43:43.909290 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.909270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:43.912168 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.912144 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 17:43:43.912376 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.912361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hck2p\"" Jul 08 17:43:43.912939 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.912917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z"] Jul 08 17:43:43.938255 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:43.938223 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln"] Jul 08 17:43:44.009198 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.009149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.009343 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.009281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962q7\" (UniqueName: \"kubernetes.io/projected/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-kube-api-access-962q7\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.009403 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.009347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk8db\" (UniqueName: \"kubernetes.io/projected/16affe09-6490-406d-8019-d0e4c7c2bcf4-kube-api-access-gk8db\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.009403 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.009395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.110213 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.110171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962q7\" (UniqueName: \"kubernetes.io/projected/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-kube-api-access-962q7\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.110405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.110225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk8db\" (UniqueName: \"kubernetes.io/projected/16affe09-6490-406d-8019-d0e4c7c2bcf4-kube-api-access-gk8db\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.110405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.110267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.110405 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.110309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.110565 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:44.110447 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 17:43:44.110565 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:44.110506 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert podName:16affe09-6490-406d-8019-d0e4c7c2bcf4 nodeName:}" failed. No retries permitted until 2026-07-08 17:43:44.610486029 +0000 UTC m=+627.412572528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert") pod "kserve-controller-manager-d6f4b65b8-fwl8z" (UID: "16affe09-6490-406d-8019-d0e4c7c2bcf4") : secret "kserve-webhook-server-cert" not found Jul 08 17:43:44.113040 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.113013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.124678 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.124652 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-962q7\" (UniqueName: \"kubernetes.io/projected/8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0-kube-api-access-962q7\") pod \"llmisvc-controller-manager-777c6bd7d5-fjpln\" (UID: \"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.124884 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.124863 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk8db\" (UniqueName: \"kubernetes.io/projected/16affe09-6490-406d-8019-d0e4c7c2bcf4-kube-api-access-gk8db\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.224394 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.224356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:44.391287 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:44.391257 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8dbe8b09_e32a_4adf_8dd2_0d53d2fa5dd0.slice/crio-e0728e5f1966ea75fa0454dc9cbfc7eb094afa0ba6c8fed50831d5c30cc43954 WatchSource:0}: Error finding container e0728e5f1966ea75fa0454dc9cbfc7eb094afa0ba6c8fed50831d5c30cc43954: Status 404 returned error can't find the container with id e0728e5f1966ea75fa0454dc9cbfc7eb094afa0ba6c8fed50831d5c30cc43954 Jul 08 17:43:44.395168 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.395146 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln"] Jul 08 17:43:44.537674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.537562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:44.542803 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.542761 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" event={"ID":"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0","Type":"ContainerStarted","Data":"e0728e5f1966ea75fa0454dc9cbfc7eb094afa0ba6c8fed50831d5c30cc43954"} Jul 08 17:43:44.615433 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.615391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.618038 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.618000 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16affe09-6490-406d-8019-d0e4c7c2bcf4-cert\") pod \"kserve-controller-manager-d6f4b65b8-fwl8z\" (UID: \"16affe09-6490-406d-8019-d0e4c7c2bcf4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.815540 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.815455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:44.980277 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:44.980251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z"] Jul 08 17:43:44.981689 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:43:44.981657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16affe09_6490_406d_8019_d0e4c7c2bcf4.slice/crio-f87254949aa5cef77a9529c577af66dfb6ed99a213a268a0d84fa49ec79397e1 WatchSource:0}: Error finding container f87254949aa5cef77a9529c577af66dfb6ed99a213a268a0d84fa49ec79397e1: Status 404 returned error can't find the container with id f87254949aa5cef77a9529c577af66dfb6ed99a213a268a0d84fa49ec79397e1 Jul 08 17:43:45.466216 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:45.466183 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:45.550369 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:45.550330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" event={"ID":"16affe09-6490-406d-8019-d0e4c7c2bcf4","Type":"ContainerStarted","Data":"f87254949aa5cef77a9529c577af66dfb6ed99a213a268a0d84fa49ec79397e1"} Jul 08 17:43:45.550526 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:45.550391 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" containerID="cri-o://bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca" gracePeriod=30 Jul 08 17:43:45.891583 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:45.891556 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:46.028960 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.028872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v64w\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.028960 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.028937 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029089 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029442 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs\") pod \"14c32628-1f6b-483a-8809-ad3e7facc637\" (UID: \"14c32628-1f6b-483a-8809-ad3e7facc637\") " Jul 08 17:43:46.029726 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.029684 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:43:46.032168 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.032133 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs" (OuterVolumeSpecName: "local-certs") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:46.032269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.032139 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts" (OuterVolumeSpecName: "cacerts") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:43:46.032269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.032219 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token" (OuterVolumeSpecName: "istio-token") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:43:46.032847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.032819 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:43:46.032967 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.032918 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:43:46.034527 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.034491 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w" (OuterVolumeSpecName: "kube-api-access-2v64w") pod "14c32628-1f6b-483a-8809-ad3e7facc637" (UID: "14c32628-1f6b-483a-8809-ad3e7facc637"). InnerVolumeSpecName "kube-api-access-2v64w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:43:46.130701 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130665 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-dns-cert\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130701 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130701 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-istio-token\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130718 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-cacerts\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130732 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14c32628-1f6b-483a-8809-ad3e7facc637-istio-csr-ca-configmap\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130746 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14c32628-1f6b-483a-8809-ad3e7facc637-istio-kubeconfig\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130758 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c32628-1f6b-483a-8809-ad3e7facc637-local-certs\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.130932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.130770 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v64w\" (UniqueName: \"kubernetes.io/projected/14c32628-1f6b-483a-8809-ad3e7facc637-kube-api-access-2v64w\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:43:46.557128 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.557093 2579 generic.go:356] "Generic (PLEG): container finished" podID="14c32628-1f6b-483a-8809-ad3e7facc637" containerID="bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca" exitCode=0 Jul 08 17:43:46.557321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.557142 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" Jul 08 17:43:46.557321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.557180 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" event={"ID":"14c32628-1f6b-483a-8809-ad3e7facc637","Type":"ContainerDied","Data":"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca"} Jul 08 17:43:46.557321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.557242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf" event={"ID":"14c32628-1f6b-483a-8809-ad3e7facc637","Type":"ContainerDied","Data":"204ad5478e886cf990799872035233250fe8a786dff35d20d44acc50181f1f3c"} Jul 08 17:43:46.557321 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.557268 2579 scope.go:117] "RemoveContainer" containerID="bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca" Jul 08 17:43:46.566304 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.566282 2579 scope.go:117] "RemoveContainer" containerID="bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca" Jul 08 17:43:46.566571 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:43:46.566554 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca\": container with ID starting with bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca not found: ID does not exist" containerID="bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca" Jul 08 17:43:46.566633 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.566582 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca"} err="failed to get container status \"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca\": rpc error: code = NotFound desc = could not find container \"bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca\": container with ID starting with bbd00bdd65797b4435c90cb2aacf3ae3112b576bf3da94bc306efc86030d06ca not found: ID does not exist" Jul 08 17:43:46.582951 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.582918 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:46.588887 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:46.588856 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ltfrf"] Jul 08 17:43:47.792951 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:47.792914 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" path="/var/lib/kubelet/pods/14c32628-1f6b-483a-8809-ad3e7facc637/volumes" Jul 08 17:43:49.514170 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.514134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r5txw" Jul 08 17:43:49.578071 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.578026 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" event={"ID":"8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0","Type":"ContainerStarted","Data":"f8ee2ef5fb30a4b0bca9097470cab6dad2ce8c7ed9213e0a444ddd42f069de52"} Jul 08 17:43:49.579999 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.579969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" event={"ID":"16affe09-6490-406d-8019-d0e4c7c2bcf4","Type":"ContainerStarted","Data":"317ae61765bb8b46d74c87cd1b0e4e61746500cc2d9ddaafe916a0bccf48f783"} Jul 08 17:43:49.580137 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.580101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:43:49.597431 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.597377 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" podStartSLOduration=2.397880622 podStartE2EDuration="6.597360256s" totalImagesPullingTime="4.199479634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:49.595300498 +0000 UTC m=+632.397387017" watchObservedRunningTime="2026-07-08 17:43:49.597360256 +0000 UTC m=+632.399446774" Jul 08 17:43:49.614992 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:49.614942 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" podStartSLOduration=3.008807046 podStartE2EDuration="6.614924637s" totalImagesPullingTime="3.606117591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:49.614231969 +0000 UTC m=+632.416318489" watchObservedRunningTime="2026-07-08 17:43:49.614924637 +0000 UTC m=+632.417011156" Jul 08 17:43:50.585455 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:50.585420 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:43:52.535655 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:43:52.535625 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-x4f7b" Jul 08 17:44:20.590531 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:20.590497 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fwl8z" Jul 08 17:44:21.591572 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:21.591543 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fjpln" Jul 08 17:44:33.757788 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.757751 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:33.758172 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.758137 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" Jul 08 17:44:33.758172 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.758148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" Jul 08 17:44:33.758249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.758202 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c32628-1f6b-483a-8809-ad3e7facc637" containerName="discovery" Jul 08 17:44:33.760528 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.760508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:33.763669 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.763646 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vbcps\"" Jul 08 17:44:33.763782 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.763699 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 17:44:33.764628 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.764588 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 17:44:33.773834 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.773808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:33.854146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.854109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnmfl\" (UniqueName: \"kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl\") pod \"kuadrant-operator-catalog-8bmh8\" (UID: \"b4c64038-cec5-4df4-8c8f-cd637f01568d\") " pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:33.955409 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.955368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnmfl\" (UniqueName: \"kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl\") pod \"kuadrant-operator-catalog-8bmh8\" (UID: \"b4c64038-cec5-4df4-8c8f-cd637f01568d\") " pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:33.964036 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:33.964006 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnmfl\" (UniqueName: \"kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl\") pod \"kuadrant-operator-catalog-8bmh8\" (UID: \"b4c64038-cec5-4df4-8c8f-cd637f01568d\") " pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:34.070784 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.070691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:34.096898 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.096865 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:34.213735 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.213712 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:34.215577 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:44:34.215549 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c64038_cec5_4df4_8c8f_cd637f01568d.slice/crio-cb9eaaee98c2978730643cc3fdf056777354adccc3f6d90d375b71da1632fde5 WatchSource:0}: Error finding container cb9eaaee98c2978730643cc3fdf056777354adccc3f6d90d375b71da1632fde5: Status 404 returned error can't find the container with id cb9eaaee98c2978730643cc3fdf056777354adccc3f6d90d375b71da1632fde5 Jul 08 17:44:34.307694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.307658 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rj656"] Jul 08 17:44:34.310422 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.310399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:34.319824 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.319793 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rj656"] Jul 08 17:44:34.360118 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.360026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8jc4\" (UniqueName: \"kubernetes.io/projected/a47cbe8a-a1b7-41b3-8518-154cd34681f9-kube-api-access-m8jc4\") pod \"kuadrant-operator-catalog-rj656\" (UID: \"a47cbe8a-a1b7-41b3-8518-154cd34681f9\") " pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:34.461050 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.461011 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8jc4\" (UniqueName: \"kubernetes.io/projected/a47cbe8a-a1b7-41b3-8518-154cd34681f9-kube-api-access-m8jc4\") pod \"kuadrant-operator-catalog-rj656\" (UID: \"a47cbe8a-a1b7-41b3-8518-154cd34681f9\") " pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:34.470291 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.470252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8jc4\" (UniqueName: \"kubernetes.io/projected/a47cbe8a-a1b7-41b3-8518-154cd34681f9-kube-api-access-m8jc4\") pod \"kuadrant-operator-catalog-rj656\" (UID: \"a47cbe8a-a1b7-41b3-8518-154cd34681f9\") " pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:34.620649 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.620526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:34.757817 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.757722 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rj656"] Jul 08 17:44:34.760325 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:44:34.760294 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47cbe8a_a1b7_41b3_8518_154cd34681f9.slice/crio-628aa3c0c50fde5c85c82406ab97ea80b732a9a5f96508f42232787006a6a87c WatchSource:0}: Error finding container 628aa3c0c50fde5c85c82406ab97ea80b732a9a5f96508f42232787006a6a87c: Status 404 returned error can't find the container with id 628aa3c0c50fde5c85c82406ab97ea80b732a9a5f96508f42232787006a6a87c Jul 08 17:44:34.793761 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.793718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rj656" event={"ID":"a47cbe8a-a1b7-41b3-8518-154cd34681f9","Type":"ContainerStarted","Data":"628aa3c0c50fde5c85c82406ab97ea80b732a9a5f96508f42232787006a6a87c"} Jul 08 17:44:34.794906 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:34.794875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" event={"ID":"b4c64038-cec5-4df4-8c8f-cd637f01568d","Type":"ContainerStarted","Data":"cb9eaaee98c2978730643cc3fdf056777354adccc3f6d90d375b71da1632fde5"} Jul 08 17:44:36.806550 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:36.806507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rj656" event={"ID":"a47cbe8a-a1b7-41b3-8518-154cd34681f9","Type":"ContainerStarted","Data":"59d797cdf7b6b5ea142abbc50d54543182071ac16a64dc27c39671564bcbdd36"} Jul 08 17:44:36.808049 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:36.807989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" event={"ID":"b4c64038-cec5-4df4-8c8f-cd637f01568d","Type":"ContainerStarted","Data":"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7"} Jul 08 17:44:36.808049 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:36.808008 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" podUID="b4c64038-cec5-4df4-8c8f-cd637f01568d" containerName="registry-server" containerID="cri-o://76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7" gracePeriod=2 Jul 08 17:44:36.830036 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:36.829931 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rj656" podStartSLOduration=1.021047656 podStartE2EDuration="2.829913078s" totalImagesPullingTime="1.808865422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:36.829510575 +0000 UTC m=+679.631597093" watchObservedRunningTime="2026-07-08 17:44:36.829913078 +0000 UTC m=+679.631999597" Jul 08 17:44:36.846772 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:36.846528 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" podStartSLOduration=1.49613597 podStartE2EDuration="3.846512599s" totalImagesPullingTime="2.350376629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:36.845328994 +0000 UTC m=+679.647415540" watchObservedRunningTime="2026-07-08 17:44:36.846512599 +0000 UTC m=+679.648599131" Jul 08 17:44:37.064787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.064761 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:37.187367 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.187267 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnmfl\" (UniqueName: \"kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl\") pod \"b4c64038-cec5-4df4-8c8f-cd637f01568d\" (UID: \"b4c64038-cec5-4df4-8c8f-cd637f01568d\") " Jul 08 17:44:37.189638 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.189610 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl" (OuterVolumeSpecName: "kube-api-access-mnmfl") pod "b4c64038-cec5-4df4-8c8f-cd637f01568d" (UID: "b4c64038-cec5-4df4-8c8f-cd637f01568d"). InnerVolumeSpecName "kube-api-access-mnmfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:44:37.288063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.288022 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnmfl\" (UniqueName: \"kubernetes.io/projected/b4c64038-cec5-4df4-8c8f-cd637f01568d-kube-api-access-mnmfl\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:44:37.815020 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.814989 2579 generic.go:356] "Generic (PLEG): container finished" podID="b4c64038-cec5-4df4-8c8f-cd637f01568d" containerID="76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7" exitCode=0 Jul 08 17:44:37.815406 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.815050 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" Jul 08 17:44:37.815406 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.815068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" event={"ID":"b4c64038-cec5-4df4-8c8f-cd637f01568d","Type":"ContainerDied","Data":"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7"} Jul 08 17:44:37.815406 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.815116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8bmh8" event={"ID":"b4c64038-cec5-4df4-8c8f-cd637f01568d","Type":"ContainerDied","Data":"cb9eaaee98c2978730643cc3fdf056777354adccc3f6d90d375b71da1632fde5"} Jul 08 17:44:37.815406 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.815135 2579 scope.go:117] "RemoveContainer" containerID="76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7" Jul 08 17:44:37.824881 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.824863 2579 scope.go:117] "RemoveContainer" containerID="76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7" Jul 08 17:44:37.825165 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:44:37.825145 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7\": container with ID starting with 76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7 not found: ID does not exist" containerID="76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7" Jul 08 17:44:37.825249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.825176 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7"} err="failed to get container status \"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7\": rpc error: code = NotFound desc = could not find container \"76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7\": container with ID starting with 76e2d0272b2dd319ee2c8a9dd6e4a6ad32feb0078dbcf4e4fd87b9def7bc29c7 not found: ID does not exist" Jul 08 17:44:37.834528 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.834499 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:37.842929 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:37.842872 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8bmh8"] Jul 08 17:44:39.791395 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:39.791364 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c64038-cec5-4df4-8c8f-cd637f01568d" path="/var/lib/kubelet/pods/b4c64038-cec5-4df4-8c8f-cd637f01568d/volumes" Jul 08 17:44:44.621206 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:44.621148 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:44.621206 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:44.621227 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:44.644855 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:44.644830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:44:44.869850 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:44:44.869823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rj656" Jul 08 17:45:05.834430 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.834385 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg"] Jul 08 17:45:05.835294 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.835269 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c64038-cec5-4df4-8c8f-cd637f01568d" containerName="registry-server" Jul 08 17:45:05.835392 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.835297 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c64038-cec5-4df4-8c8f-cd637f01568d" containerName="registry-server" Jul 08 17:45:05.835506 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.835489 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c64038-cec5-4df4-8c8f-cd637f01568d" containerName="registry-server" Jul 08 17:45:05.843748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.843718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:05.847682 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.847657 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fgfbh\"" Jul 08 17:45:05.854049 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.854025 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg"] Jul 08 17:45:05.932119 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:05.932072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzdnr\" (UniqueName: \"kubernetes.io/projected/5b021dc6-60bc-45de-bd13-f7872a635a49-kube-api-access-hzdnr\") pod \"authorino-operator-75cc8d5cb7-jjdgg\" (UID: \"5b021dc6-60bc-45de-bd13-f7872a635a49\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:06.033180 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:06.033139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzdnr\" (UniqueName: \"kubernetes.io/projected/5b021dc6-60bc-45de-bd13-f7872a635a49-kube-api-access-hzdnr\") pod \"authorino-operator-75cc8d5cb7-jjdgg\" (UID: \"5b021dc6-60bc-45de-bd13-f7872a635a49\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:06.043124 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:06.043092 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzdnr\" (UniqueName: \"kubernetes.io/projected/5b021dc6-60bc-45de-bd13-f7872a635a49-kube-api-access-hzdnr\") pod \"authorino-operator-75cc8d5cb7-jjdgg\" (UID: \"5b021dc6-60bc-45de-bd13-f7872a635a49\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:06.154698 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:06.154569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:06.291542 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:06.291367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg"] Jul 08 17:45:06.294401 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:06.294375 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b021dc6_60bc_45de_bd13_f7872a635a49.slice/crio-5e3840003d16a952913450872fb1fcf1dec0563eec66f180bee502ee7c4cbf35 WatchSource:0}: Error finding container 5e3840003d16a952913450872fb1fcf1dec0563eec66f180bee502ee7c4cbf35: Status 404 returned error can't find the container with id 5e3840003d16a952913450872fb1fcf1dec0563eec66f180bee502ee7c4cbf35 Jul 08 17:45:06.955345 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:06.953762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" event={"ID":"5b021dc6-60bc-45de-bd13-f7872a635a49","Type":"ContainerStarted","Data":"5e3840003d16a952913450872fb1fcf1dec0563eec66f180bee502ee7c4cbf35"} Jul 08 17:45:09.012067 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.011968 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885"] Jul 08 17:45:09.015997 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.015968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:09.021574 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.021550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mt4hl\"" Jul 08 17:45:09.032338 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.032314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885"] Jul 08 17:45:09.053674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.053640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b62xw\" (UniqueName: \"kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw\") pod \"limitador-operator-controller-manager-859d7b78bd-54885\" (UID: \"ca83da56-00b8-47f9-b12d-50e83c27f6ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:09.155119 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.155064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b62xw\" (UniqueName: \"kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw\") pod \"limitador-operator-controller-manager-859d7b78bd-54885\" (UID: \"ca83da56-00b8-47f9-b12d-50e83c27f6ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:09.168446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.168412 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b62xw\" (UniqueName: \"kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw\") pod \"limitador-operator-controller-manager-859d7b78bd-54885\" (UID: \"ca83da56-00b8-47f9-b12d-50e83c27f6ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:09.329374 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.329334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:09.490692 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.490566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885"] Jul 08 17:45:09.970865 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.970827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" event={"ID":"5b021dc6-60bc-45de-bd13-f7872a635a49","Type":"ContainerStarted","Data":"25c275dd562b0322af291ce6c692063d20864a3c0d4462eda866a2af84aa7d84"} Jul 08 17:45:09.971069 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.970891 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:09.972064 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:09.972042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" event={"ID":"ca83da56-00b8-47f9-b12d-50e83c27f6ca","Type":"ContainerStarted","Data":"b5ffce969119d93592551800e69e73a714ccf60afb858e6a57561b49112e733e"} Jul 08 17:45:10.023028 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:10.022973 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" podStartSLOduration=2.069850624 podStartE2EDuration="5.022956756s" totalImagesPullingTime="2.953106132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:10.022236421 +0000 UTC m=+712.824322941" watchObservedRunningTime="2026-07-08 17:45:10.022956756 +0000 UTC m=+712.825043275" Jul 08 17:45:11.984858 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:11.984821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" event={"ID":"ca83da56-00b8-47f9-b12d-50e83c27f6ca","Type":"ContainerStarted","Data":"b344246d331af08fce1a137e3867f1533ea914924ca738ac3d9384d79348a474"} Jul 08 17:45:11.985243 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:11.984882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:12.024485 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:12.024424 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" podStartSLOduration=1.701995572 podStartE2EDuration="4.024404953s" totalImagesPullingTime="2.322409381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:12.022066872 +0000 UTC m=+714.824153392" watchObservedRunningTime="2026-07-08 17:45:12.024404953 +0000 UTC m=+714.826491473" Jul 08 17:45:13.023638 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.023591 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56"] Jul 08 17:45:13.027263 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.027239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.030081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.030062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-799hk\"" Jul 08 17:45:13.040975 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.040949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56"] Jul 08 17:45:13.091922 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.091881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5l6\" (UniqueName: \"kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.092120 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.091970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.192589 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.192548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5l6\" (UniqueName: \"kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.192811 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.192642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.193078 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.193060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.202952 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.202927 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5l6\" (UniqueName: \"kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.339098 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.339008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:13.482167 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.482137 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56"] Jul 08 17:45:13.484969 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:13.484928 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ec9654_09e9_46d0_9e6b_23cb2a0e7665.slice/crio-cdf6a47cf857c73f4349949ee595cd3f0b5a7c11487a3895b2b424aefcd420fc WatchSource:0}: Error finding container cdf6a47cf857c73f4349949ee595cd3f0b5a7c11487a3895b2b424aefcd420fc: Status 404 returned error can't find the container with id cdf6a47cf857c73f4349949ee595cd3f0b5a7c11487a3895b2b424aefcd420fc Jul 08 17:45:13.998622 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:13.998567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" event={"ID":"30ec9654-09e9-46d0-9e6b-23cb2a0e7665","Type":"ContainerStarted","Data":"cdf6a47cf857c73f4349949ee595cd3f0b5a7c11487a3895b2b424aefcd420fc"} Jul 08 17:45:20.979113 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:20.979084 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jjdgg" Jul 08 17:45:21.045063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:21.045018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" event={"ID":"30ec9654-09e9-46d0-9e6b-23cb2a0e7665","Type":"ContainerStarted","Data":"c879e0377c6cfa4d920fd7bcfe809f322e94b05931bd92599b5cf73045d0d25b"} Jul 08 17:45:21.045251 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:21.045168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:21.066285 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:21.066227 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" podStartSLOduration=2.557973332 podStartE2EDuration="9.066210168s" totalImagesPullingTime="6.508236836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:21.06529888 +0000 UTC m=+723.867385400" watchObservedRunningTime="2026-07-08 17:45:21.066210168 +0000 UTC m=+723.868296686" Jul 08 17:45:22.993675 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:22.993641 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:32.052750 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:32.052723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:33.081524 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.081489 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g"] Jul 08 17:45:33.085674 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.085656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.101188 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.101155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g"] Jul 08 17:45:33.168910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.168854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zpf\" (UniqueName: \"kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.168910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.168912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.270246 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.270203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zpf\" (UniqueName: \"kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.270246 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.270248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.270576 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.270559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.287585 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.287556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zpf\" (UniqueName: \"kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.396662 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.396542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:33.539189 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.539164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g"] Jul 08 17:45:33.541545 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:33.541515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231e6130_fbbf_4446_8d88_7b2ca8d989d4.slice/crio-77f02ab86b771aa201e6ce6cfbfac36ee7688b52a57eeabb70a6b3e86512b7de WatchSource:0}: Error finding container 77f02ab86b771aa201e6ce6cfbfac36ee7688b52a57eeabb70a6b3e86512b7de: Status 404 returned error can't find the container with id 77f02ab86b771aa201e6ce6cfbfac36ee7688b52a57eeabb70a6b3e86512b7de Jul 08 17:45:33.822541 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.822506 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g"] Jul 08 17:45:33.828446 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.828417 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g"] Jul 08 17:45:33.845968 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.845940 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56"] Jul 08 17:45:33.846275 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.846226 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" containerName="manager" containerID="cri-o://c879e0377c6cfa4d920fd7bcfe809f322e94b05931bd92599b5cf73045d0d25b" gracePeriod=2 Jul 08 17:45:33.858871 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.858845 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:33.859224 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.859209 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" containerName="manager" Jul 08 17:45:33.859271 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.859225 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" containerName="manager" Jul 08 17:45:33.859311 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.859300 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" containerName="manager" Jul 08 17:45:33.862128 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:33.862097 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-fqh56\": pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:33.862312 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.862296 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56"] Jul 08 17:45:33.862394 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.862382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:33.872021 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.871979 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:33.872344 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.872330 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" containerName="manager" Jul 08 17:45:33.872344 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.872343 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" containerName="manager" Jul 08 17:45:33.872500 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.872409 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" containerName="manager" Jul 08 17:45:33.876509 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.876490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:33.877847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.877805 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885"] Jul 08 17:45:33.878075 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.878039 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" containerName="manager" containerID="cri-o://b344246d331af08fce1a137e3867f1533ea914924ca738ac3d9384d79348a474" gracePeriod=2 Jul 08 17:45:33.880491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.880459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:33.888303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.888270 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885"] Jul 08 17:45:33.896809 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.896781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:33.905916 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:33.905637 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:33.907818 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:33.907788 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:33.932704 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:33.932662 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:33.976306 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.976272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjbh\" (UniqueName: \"kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:33.976438 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.976383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l8w\" (UniqueName: \"kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:33.976487 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.976445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:33.976487 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:33.976482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.077570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.077354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjbh\" (UniqueName: \"kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:34.077570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.077444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l8w\" (UniqueName: \"kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.077570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.077515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:34.077570 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.077561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.078275 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.077998 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.078275 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.078008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:34.094112 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.094080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l8w\" (UniqueName: \"kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-b469n\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.096404 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.096380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjbh\" (UniqueName: \"kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ntsmf\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:34.111316 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.111286 2579 generic.go:356] "Generic (PLEG): container finished" podID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" containerID="c879e0377c6cfa4d920fd7bcfe809f322e94b05931bd92599b5cf73045d0d25b" exitCode=0 Jul 08 17:45:34.113180 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.113159 2579 generic.go:356] "Generic (PLEG): container finished" podID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" containerID="b344246d331af08fce1a137e3867f1533ea914924ca738ac3d9384d79348a474" exitCode=0 Jul 08 17:45:34.115309 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.115288 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" container="manager" Jul 08 17:45:34.117346 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.117323 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:34.119197 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.119169 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:34.119581 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.119560 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:34.121138 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.121116 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:34.123112 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.122928 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:34.124846 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.124820 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:34.126855 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.126815 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:34.130808 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.130783 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:34.132964 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.132937 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:34.135736 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.135708 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:34.137657 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:34.137634 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:34.178923 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.178894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z5l6\" (UniqueName: \"kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6\") pod \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " Jul 08 17:45:34.179081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.179059 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume\") pod \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\" (UID: \"30ec9654-09e9-46d0-9e6b-23cb2a0e7665\") " Jul 08 17:45:34.179505 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.179483 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "30ec9654-09e9-46d0-9e6b-23cb2a0e7665" (UID: "30ec9654-09e9-46d0-9e6b-23cb2a0e7665"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:34.181195 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.181169 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6" (OuterVolumeSpecName: "kube-api-access-7z5l6") pod "30ec9654-09e9-46d0-9e6b-23cb2a0e7665" (UID: "30ec9654-09e9-46d0-9e6b-23cb2a0e7665"). InnerVolumeSpecName "kube-api-access-7z5l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:34.273159 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.273120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:34.279283 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.279259 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:34.279565 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.279546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b62xw\" (UniqueName: \"kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw\") pod \"ca83da56-00b8-47f9-b12d-50e83c27f6ca\" (UID: \"ca83da56-00b8-47f9-b12d-50e83c27f6ca\") " Jul 08 17:45:34.279733 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.279722 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:34.279776 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.279735 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z5l6\" (UniqueName: \"kubernetes.io/projected/30ec9654-09e9-46d0-9e6b-23cb2a0e7665-kube-api-access-7z5l6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:34.281958 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.281930 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw" (OuterVolumeSpecName: "kube-api-access-b62xw") pod "ca83da56-00b8-47f9-b12d-50e83c27f6ca" (UID: "ca83da56-00b8-47f9-b12d-50e83c27f6ca"). InnerVolumeSpecName "kube-api-access-b62xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:34.380746 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.380711 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b62xw\" (UniqueName: \"kubernetes.io/projected/ca83da56-00b8-47f9-b12d-50e83c27f6ca-kube-api-access-b62xw\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:34.431476 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.431444 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:34.434075 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:34.434045 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97842237_17f5_4c82_baeb_66f7a75f5470.slice/crio-490a021458d60f8b05452c55838d3e80dc707daeb42d35bf3bfc28cf6e010e84 WatchSource:0}: Error finding container 490a021458d60f8b05452c55838d3e80dc707daeb42d35bf3bfc28cf6e010e84: Status 404 returned error can't find the container with id 490a021458d60f8b05452c55838d3e80dc707daeb42d35bf3bfc28cf6e010e84 Jul 08 17:45:34.455820 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:34.455786 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:34.459539 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:34.459512 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d5b5e9_bac0_41f5_bca0_bcf721870b23.slice/crio-ae75fb70d5e435c383f3311baf771181c3d8765aa997982123b74378318b8c08 WatchSource:0}: Error finding container ae75fb70d5e435c383f3311baf771181c3d8765aa997982123b74378318b8c08: Status 404 returned error can't find the container with id ae75fb70d5e435c383f3311baf771181c3d8765aa997982123b74378318b8c08 Jul 08 17:45:35.121792 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.121762 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:35.121792 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.121791 2579 scope.go:117] "RemoveContainer" containerID="c879e0377c6cfa4d920fd7bcfe809f322e94b05931bd92599b5cf73045d0d25b" Jul 08 17:45:35.123879 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.123848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" event={"ID":"a5d5b5e9-bac0-41f5-bca0-bcf721870b23","Type":"ContainerStarted","Data":"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae"} Jul 08 17:45:35.124021 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.123903 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" event={"ID":"a5d5b5e9-bac0-41f5-bca0-bcf721870b23","Type":"ContainerStarted","Data":"ae75fb70d5e435c383f3311baf771181c3d8765aa997982123b74378318b8c08"} Jul 08 17:45:35.124021 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.123925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:35.124435 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.124407 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:35.125318 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.125296 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:35.126730 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.126699 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:35.127229 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.127202 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" containerName="manager" containerID="cri-o://bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc" gracePeriod=2 Jul 08 17:45:35.128924 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.128742 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:35.129179 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.129156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" event={"ID":"97842237-17f5-4c82-baeb-66f7a75f5470","Type":"ContainerStarted","Data":"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab"} Jul 08 17:45:35.129269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.129205 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" event={"ID":"97842237-17f5-4c82-baeb-66f7a75f5470","Type":"ContainerStarted","Data":"490a021458d60f8b05452c55838d3e80dc707daeb42d35bf3bfc28cf6e010e84"} Jul 08 17:45:35.129269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.129248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:35.132806 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.132776 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:35.133263 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.133241 2579 scope.go:117] "RemoveContainer" containerID="b344246d331af08fce1a137e3867f1533ea914924ca738ac3d9384d79348a474" Jul 08 17:45:35.134942 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.134916 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:35.173936 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.173871 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" podStartSLOduration=2.173847561 podStartE2EDuration="2.173847561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:35.172726373 +0000 UTC m=+737.974812892" watchObservedRunningTime="2026-07-08 17:45:35.173847561 +0000 UTC m=+737.975934081" Jul 08 17:45:35.175090 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.175059 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:35.177005 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.176983 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:35.186357 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.186321 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:35.221299 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.221243 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" podStartSLOduration=2.221227628 podStartE2EDuration="2.221227628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:35.2196542 +0000 UTC m=+738.021740765" watchObservedRunningTime="2026-07-08 17:45:35.221227628 +0000 UTC m=+738.023314146" Jul 08 17:45:35.222141 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.222112 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:35.375283 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.375226 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:35.377634 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.377583 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-fqh56\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-fqh56" Jul 08 17:45:35.379412 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.379391 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:35.381057 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:35.381035 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-54885\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-54885" Jul 08 17:45:35.491666 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.491624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8zpf\" (UniqueName: \"kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf\") pod \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " Jul 08 17:45:35.491810 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.491717 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume\") pod \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\" (UID: \"231e6130-fbbf-4446-8d88-7b2ca8d989d4\") " Jul 08 17:45:35.492016 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.491994 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "231e6130-fbbf-4446-8d88-7b2ca8d989d4" (UID: "231e6130-fbbf-4446-8d88-7b2ca8d989d4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.493800 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.493776 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf" (OuterVolumeSpecName: "kube-api-access-b8zpf") pod "231e6130-fbbf-4446-8d88-7b2ca8d989d4" (UID: "231e6130-fbbf-4446-8d88-7b2ca8d989d4"). InnerVolumeSpecName "kube-api-access-b8zpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:35.592900 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.592866 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/231e6130-fbbf-4446-8d88-7b2ca8d989d4-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.592900 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.592903 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zpf\" (UniqueName: \"kubernetes.io/projected/231e6130-fbbf-4446-8d88-7b2ca8d989d4-kube-api-access-b8zpf\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.797948 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.797910 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" path="/var/lib/kubelet/pods/231e6130-fbbf-4446-8d88-7b2ca8d989d4/volumes" Jul 08 17:45:35.798269 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.798255 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ec9654-09e9-46d0-9e6b-23cb2a0e7665" path="/var/lib/kubelet/pods/30ec9654-09e9-46d0-9e6b-23cb2a0e7665/volumes" Jul 08 17:45:35.798534 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:35.798523 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" path="/var/lib/kubelet/pods/ca83da56-00b8-47f9-b12d-50e83c27f6ca/volumes" Jul 08 17:45:36.136227 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:36.136142 2579 generic.go:356] "Generic (PLEG): container finished" podID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" containerID="bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc" exitCode=0 Jul 08 17:45:36.136227 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:36.136187 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:36.136740 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:36.136231 2579 scope.go:117] "RemoveContainer" containerID="bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc" Jul 08 17:45:36.141419 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:36.141387 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:36.145925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:36.145903 2579 scope.go:117] "RemoveContainer" containerID="bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc" Jul 08 17:45:36.146260 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:36.146240 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc\": container with ID starting with bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc not found: ID does not exist" containerID="bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc" Jul 08 17:45:36.146310 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:36.146271 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc"} err="failed to get container status \"bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc\": rpc error: code = NotFound desc = could not find container \"bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc\": container with ID starting with bd350ac0cbb2c3731c38a7e81ccfe6a98d6d2aad2a8665b40c75058bcda1dbfc not found: ID does not exist" Jul 08 17:45:37.789267 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:37.789231 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6hv9g\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="231e6130-fbbf-4446-8d88-7b2ca8d989d4" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6hv9g" Jul 08 17:45:46.140709 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.140670 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:46.141063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.140742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:46.222721 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.222689 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:46.222926 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.222901 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" podUID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" containerName="manager" containerID="cri-o://bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae" gracePeriod=10 Jul 08 17:45:46.423229 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.423195 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:45:46.423575 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.423561 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" containerName="manager" Jul 08 17:45:46.423575 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.423574 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" containerName="manager" Jul 08 17:45:46.423746 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.423669 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca83da56-00b8-47f9-b12d-50e83c27f6ca" containerName="manager" Jul 08 17:45:46.425670 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.425649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.445041 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.445015 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:45:46.485924 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.485891 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:46.598470 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.598435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9l8w\" (UniqueName: \"kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w\") pod \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " Jul 08 17:45:46.598672 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.598484 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume\") pod \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\" (UID: \"a5d5b5e9-bac0-41f5-bca0-bcf721870b23\") " Jul 08 17:45:46.598734 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.598709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glvpr\" (UniqueName: \"kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.598915 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.598887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.598915 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.598912 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5d5b5e9-bac0-41f5-bca0-bcf721870b23" (UID: "a5d5b5e9-bac0-41f5-bca0-bcf721870b23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:46.599078 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.599007 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:46.600819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.600799 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w" (OuterVolumeSpecName: "kube-api-access-g9l8w") pod "a5d5b5e9-bac0-41f5-bca0-bcf721870b23" (UID: "a5d5b5e9-bac0-41f5-bca0-bcf721870b23"). InnerVolumeSpecName "kube-api-access-g9l8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:46.699700 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.699644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glvpr\" (UniqueName: \"kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.699879 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.699740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.699879 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.699786 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9l8w\" (UniqueName: \"kubernetes.io/projected/a5d5b5e9-bac0-41f5-bca0-bcf721870b23-kube-api-access-g9l8w\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:46.700104 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.700088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.711376 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.711342 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glvpr\" (UniqueName: \"kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr\") pod \"kuadrant-operator-controller-manager-86878c769f-96gsp\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.736234 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.736203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:46.877846 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:46.877807 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:45:46.880214 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:45:46.880188 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5283e5bd_0c59_4173_92b9_bc433cd708d5.slice/crio-1b58ecf3589de6857db617f458bd2716f7b3fff5e5267dd0e9b7ced82b9c39d7 WatchSource:0}: Error finding container 1b58ecf3589de6857db617f458bd2716f7b3fff5e5267dd0e9b7ced82b9c39d7: Status 404 returned error can't find the container with id 1b58ecf3589de6857db617f458bd2716f7b3fff5e5267dd0e9b7ced82b9c39d7 Jul 08 17:45:47.195864 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.195830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" event={"ID":"5283e5bd-0c59-4173-92b9-bc433cd708d5","Type":"ContainerStarted","Data":"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265"} Jul 08 17:45:47.196303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.195884 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" event={"ID":"5283e5bd-0c59-4173-92b9-bc433cd708d5","Type":"ContainerStarted","Data":"1b58ecf3589de6857db617f458bd2716f7b3fff5e5267dd0e9b7ced82b9c39d7"} Jul 08 17:45:47.196303 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.195981 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:47.197133 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.197113 2579 generic.go:356] "Generic (PLEG): container finished" podID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" containerID="bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae" exitCode=0 Jul 08 17:45:47.197230 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.197151 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" Jul 08 17:45:47.197230 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.197191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" event={"ID":"a5d5b5e9-bac0-41f5-bca0-bcf721870b23","Type":"ContainerDied","Data":"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae"} Jul 08 17:45:47.197323 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.197242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n" event={"ID":"a5d5b5e9-bac0-41f5-bca0-bcf721870b23","Type":"ContainerDied","Data":"ae75fb70d5e435c383f3311baf771181c3d8765aa997982123b74378318b8c08"} Jul 08 17:45:47.197323 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.197269 2579 scope.go:117] "RemoveContainer" containerID="bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae" Jul 08 17:45:47.207491 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.207473 2579 scope.go:117] "RemoveContainer" containerID="bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae" Jul 08 17:45:47.207867 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:47.207848 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae\": container with ID starting with bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae not found: ID does not exist" containerID="bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae" Jul 08 17:45:47.207922 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.207878 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae"} err="failed to get container status \"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae\": rpc error: code = NotFound desc = could not find container \"bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae\": container with ID starting with bdde928fd5efb05f1f779827e3fc15df4856b88cb2dd1c8cd2412d166a183cae not found: ID does not exist" Jul 08 17:45:47.221589 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.221532 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" podStartSLOduration=1.221515041 podStartE2EDuration="1.221515041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:47.219572824 +0000 UTC m=+750.021659343" watchObservedRunningTime="2026-07-08 17:45:47.221515041 +0000 UTC m=+750.023601560" Jul 08 17:45:47.239856 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.239829 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:47.245535 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.245509 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-b469n"] Jul 08 17:45:47.791472 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:47.791437 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" path="/var/lib/kubelet/pods/a5d5b5e9-bac0-41f5-bca0-bcf721870b23/volumes" Jul 08 17:45:58.206064 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.206028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:45:58.296561 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.296532 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:58.296786 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.296764 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" podUID="97842237-17f5-4c82-baeb-66f7a75f5470" containerName="manager" containerID="cri-o://f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab" gracePeriod=10 Jul 08 17:45:58.565450 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.565424 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:58.598902 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.598860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume\") pod \"97842237-17f5-4c82-baeb-66f7a75f5470\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " Jul 08 17:45:58.599070 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.598957 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pjbh\" (UniqueName: \"kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh\") pod \"97842237-17f5-4c82-baeb-66f7a75f5470\" (UID: \"97842237-17f5-4c82-baeb-66f7a75f5470\") " Jul 08 17:45:58.599280 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.599254 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97842237-17f5-4c82-baeb-66f7a75f5470" (UID: "97842237-17f5-4c82-baeb-66f7a75f5470"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:58.601658 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.601622 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh" (OuterVolumeSpecName: "kube-api-access-9pjbh") pod "97842237-17f5-4c82-baeb-66f7a75f5470" (UID: "97842237-17f5-4c82-baeb-66f7a75f5470"). InnerVolumeSpecName "kube-api-access-9pjbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:58.699815 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.699772 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97842237-17f5-4c82-baeb-66f7a75f5470-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:58.699815 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:58.699807 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pjbh\" (UniqueName: \"kubernetes.io/projected/97842237-17f5-4c82-baeb-66f7a75f5470-kube-api-access-9pjbh\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:45:59.260236 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.260200 2579 generic.go:356] "Generic (PLEG): container finished" podID="97842237-17f5-4c82-baeb-66f7a75f5470" containerID="f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab" exitCode=0 Jul 08 17:45:59.260728 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.260240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" event={"ID":"97842237-17f5-4c82-baeb-66f7a75f5470","Type":"ContainerDied","Data":"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab"} Jul 08 17:45:59.260728 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.260290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" event={"ID":"97842237-17f5-4c82-baeb-66f7a75f5470","Type":"ContainerDied","Data":"490a021458d60f8b05452c55838d3e80dc707daeb42d35bf3bfc28cf6e010e84"} Jul 08 17:45:59.260728 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.260318 2579 scope.go:117] "RemoveContainer" containerID="f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab" Jul 08 17:45:59.260728 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.260245 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf" Jul 08 17:45:59.270280 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.270263 2579 scope.go:117] "RemoveContainer" containerID="f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab" Jul 08 17:45:59.270576 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:45:59.270557 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab\": container with ID starting with f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab not found: ID does not exist" containerID="f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab" Jul 08 17:45:59.270648 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.270587 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab"} err="failed to get container status \"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab\": rpc error: code = NotFound desc = could not find container \"f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab\": container with ID starting with f91b25d9329e5ae32a47fd75098426428bd34b0c2dcdd4cfca1876a99603b1ab not found: ID does not exist" Jul 08 17:45:59.288048 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.288013 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:59.307823 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.307791 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ntsmf"] Jul 08 17:45:59.791692 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:45:59.791662 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97842237-17f5-4c82-baeb-66f7a75f5470" path="/var/lib/kubelet/pods/97842237-17f5-4c82-baeb-66f7a75f5470/volumes" Jul 08 17:46:15.224251 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224212 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224621 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" containerName="manager" Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224634 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" containerName="manager" Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224644 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97842237-17f5-4c82-baeb-66f7a75f5470" containerName="manager" Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224649 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97842237-17f5-4c82-baeb-66f7a75f5470" containerName="manager" Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224710 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d5b5e9-bac0-41f5-bca0-bcf721870b23" containerName="manager" Jul 08 17:46:15.224748 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.224720 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97842237-17f5-4c82-baeb-66f7a75f5470" containerName="manager" Jul 08 17:46:15.227665 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.227647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.230333 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.230309 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slgqj\"" Jul 08 17:46:15.230471 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.230315 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 17:46:15.238153 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.238132 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:15.322061 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.322023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:15.353791 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.353760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5k6\" (UniqueName: \"kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.353957 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.353843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.454520 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.454476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5k6\" (UniqueName: \"kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.454741 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.454544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.455189 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.455168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.465507 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.465480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5k6\" (UniqueName: \"kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6\") pod \"limitador-limitador-64c94fd64f-2c5ms\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.528765 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.528675 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:46:15.532404 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.532382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.538241 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.538214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:15.540870 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.540834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:46:15.584929 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.584894 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:46:15.657473 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.657435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-config-file\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.657677 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.657484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hnw\" (UniqueName: \"kubernetes.io/projected/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-kube-api-access-z9hnw\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.691819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.691791 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:15.693742 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:46:15.693716 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5a05cc_d18f_44e3_a7ff_7f606a05c852.slice/crio-a16c4aae3f8f205e061223ad7419b8e5f3ce59bd8870230787f54be89f299b12 WatchSource:0}: Error finding container a16c4aae3f8f205e061223ad7419b8e5f3ce59bd8870230787f54be89f299b12: Status 404 returned error can't find the container with id a16c4aae3f8f205e061223ad7419b8e5f3ce59bd8870230787f54be89f299b12 Jul 08 17:46:15.758866 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.758822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-config-file\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.758866 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.758860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hnw\" (UniqueName: \"kubernetes.io/projected/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-kube-api-access-z9hnw\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.759462 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.759443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-config-file\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.767355 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.767323 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hnw\" (UniqueName: \"kubernetes.io/projected/dee51a9c-58e0-4d02-8e8e-74a9beac6c6b-kube-api-access-z9hnw\") pod \"limitador-limitador-54d6d988d5-72t8w\" (UID: \"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.844400 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.844306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:15.982904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:15.982877 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:46:15.984910 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:46:15.984881 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee51a9c_58e0_4d02_8e8e_74a9beac6c6b.slice/crio-bcd8011da3d747ccb2c669fcf81c7be03ddad9a6e7689d5b398878edfc9e70e8 WatchSource:0}: Error finding container bcd8011da3d747ccb2c669fcf81c7be03ddad9a6e7689d5b398878edfc9e70e8: Status 404 returned error can't find the container with id bcd8011da3d747ccb2c669fcf81c7be03ddad9a6e7689d5b398878edfc9e70e8 Jul 08 17:46:16.347586 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:16.347546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" event={"ID":"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b","Type":"ContainerStarted","Data":"bcd8011da3d747ccb2c669fcf81c7be03ddad9a6e7689d5b398878edfc9e70e8"} Jul 08 17:46:16.348736 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:16.348703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" event={"ID":"8f5a05cc-d18f-44e3-a7ff-7f606a05c852","Type":"ContainerStarted","Data":"a16c4aae3f8f205e061223ad7419b8e5f3ce59bd8870230787f54be89f299b12"} Jul 08 17:46:19.369833 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.369796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" event={"ID":"dee51a9c-58e0-4d02-8e8e-74a9beac6c6b","Type":"ContainerStarted","Data":"bc9595273bf3acb19b3faa87fdbe2eb769deec54daeeae16f01d5fc6a38e643f"} Jul 08 17:46:19.370277 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.369923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:19.371526 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.371503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" event={"ID":"8f5a05cc-d18f-44e3-a7ff-7f606a05c852","Type":"ContainerStarted","Data":"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb"} Jul 08 17:46:19.371673 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.371558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:19.389260 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.389211 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" podStartSLOduration=1.563019433 podStartE2EDuration="4.38919738s" totalImagesPullingTime="2.826177947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:19.38804941 +0000 UTC m=+782.190135929" watchObservedRunningTime="2026-07-08 17:46:19.38919738 +0000 UTC m=+782.191283898" Jul 08 17:46:19.404528 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:19.404482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" podStartSLOduration=1.279452886 podStartE2EDuration="4.404468251s" totalImagesPullingTime="3.125015365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:19.404257835 +0000 UTC m=+782.206344353" watchObservedRunningTime="2026-07-08 17:46:19.404468251 +0000 UTC m=+782.206554780" Jul 08 17:46:30.377925 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:30.377885 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-72t8w" Jul 08 17:46:30.378320 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:30.377960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:30.453296 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:30.453257 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:30.453487 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:30.453441 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" podUID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" containerName="limitador" containerID="cri-o://c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb" gracePeriod=30 Jul 08 17:46:31.010057 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.010024 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:31.098757 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.098658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z5k6\" (UniqueName: \"kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6\") pod \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " Jul 08 17:46:31.098757 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.098726 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file\") pod \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\" (UID: \"8f5a05cc-d18f-44e3-a7ff-7f606a05c852\") " Jul 08 17:46:31.099069 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.099036 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file" (OuterVolumeSpecName: "config-file") pod "8f5a05cc-d18f-44e3-a7ff-7f606a05c852" (UID: "8f5a05cc-d18f-44e3-a7ff-7f606a05c852"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:46:31.100910 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.100884 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6" (OuterVolumeSpecName: "kube-api-access-2z5k6") pod "8f5a05cc-d18f-44e3-a7ff-7f606a05c852" (UID: "8f5a05cc-d18f-44e3-a7ff-7f606a05c852"). InnerVolumeSpecName "kube-api-access-2z5k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:31.199384 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.199344 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z5k6\" (UniqueName: \"kubernetes.io/projected/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-kube-api-access-2z5k6\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:46:31.199384 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.199381 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f5a05cc-d18f-44e3-a7ff-7f606a05c852-config-file\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:46:31.430988 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.430891 2579 generic.go:356] "Generic (PLEG): container finished" podID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" containerID="c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb" exitCode=0 Jul 08 17:46:31.430988 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.430936 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" Jul 08 17:46:31.430988 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.430976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" event={"ID":"8f5a05cc-d18f-44e3-a7ff-7f606a05c852","Type":"ContainerDied","Data":"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb"} Jul 08 17:46:31.431503 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.431041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2c5ms" event={"ID":"8f5a05cc-d18f-44e3-a7ff-7f606a05c852","Type":"ContainerDied","Data":"a16c4aae3f8f205e061223ad7419b8e5f3ce59bd8870230787f54be89f299b12"} Jul 08 17:46:31.431503 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.431076 2579 scope.go:117] "RemoveContainer" containerID="c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb" Jul 08 17:46:31.440786 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.440765 2579 scope.go:117] "RemoveContainer" containerID="c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb" Jul 08 17:46:31.441165 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:46:31.441143 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb\": container with ID starting with c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb not found: ID does not exist" containerID="c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb" Jul 08 17:46:31.441225 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.441176 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb"} err="failed to get container status \"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb\": rpc error: code = NotFound desc = could not find container \"c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb\": container with ID starting with c654f6651466c67880effec9fe2f3af3d9a41902da76abb833887cb841e363cb not found: ID does not exist" Jul 08 17:46:31.453759 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.453728 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:31.458040 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.458000 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2c5ms"] Jul 08 17:46:31.791862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:31.791831 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" path="/var/lib/kubelet/pods/8f5a05cc-d18f-44e3-a7ff-7f606a05c852/volumes" Jul 08 17:46:32.139424 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.139349 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw"] Jul 08 17:46:32.139757 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.139743 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" containerName="limitador" Jul 08 17:46:32.139808 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.139758 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" containerName="limitador" Jul 08 17:46:32.139847 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.139826 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5a05cc-d18f-44e3-a7ff-7f606a05c852" containerName="limitador" Jul 08 17:46:32.144127 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.144109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.147120 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.147094 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tqlqn\"" Jul 08 17:46:32.147336 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.147316 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 17:46:32.147968 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.147950 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 17:46:32.148032 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.147987 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 17:46:32.153109 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.153086 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw"] Jul 08 17:46:32.308732 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.308690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdw2c\" (UniqueName: \"kubernetes.io/projected/0af962eb-a9e3-4476-a690-29967c179470-kube-api-access-zdw2c\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.308732 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.308738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0af962eb-a9e3-4476-a690-29967c179470-data\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.410161 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.410056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdw2c\" (UniqueName: \"kubernetes.io/projected/0af962eb-a9e3-4476-a690-29967c179470-kube-api-access-zdw2c\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.410161 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.410105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0af962eb-a9e3-4476-a690-29967c179470-data\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.410479 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.410461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0af962eb-a9e3-4476-a690-29967c179470-data\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.421100 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.421071 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdw2c\" (UniqueName: \"kubernetes.io/projected/0af962eb-a9e3-4476-a690-29967c179470-kube-api-access-zdw2c\") pod \"postgres-69fb4594cf-b4hpw\" (UID: \"0af962eb-a9e3-4476-a690-29967c179470\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.455352 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.455314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:32.601928 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:32.601888 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw"] Jul 08 17:46:32.604340 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:46:32.604306 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af962eb_a9e3_4476_a690_29967c179470.slice/crio-37e09954e2cb691be1e8731de21ed122f6a131e2c2249650ba1800ff428b77b8 WatchSource:0}: Error finding container 37e09954e2cb691be1e8731de21ed122f6a131e2c2249650ba1800ff428b77b8: Status 404 returned error can't find the container with id 37e09954e2cb691be1e8731de21ed122f6a131e2c2249650ba1800ff428b77b8 Jul 08 17:46:33.446804 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:33.446764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" event={"ID":"0af962eb-a9e3-4476-a690-29967c179470","Type":"ContainerStarted","Data":"37e09954e2cb691be1e8731de21ed122f6a131e2c2249650ba1800ff428b77b8"} Jul 08 17:46:38.477142 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:38.477102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" event={"ID":"0af962eb-a9e3-4476-a690-29967c179470","Type":"ContainerStarted","Data":"9286c0b8bba12b0d4f9ffdbca4f966fb5419f2c024b98e72f1c4f2db588b81d5"} Jul 08 17:46:38.477649 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:38.477190 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:38.493979 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:38.493923 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" podStartSLOduration=1.562725591 podStartE2EDuration="6.493902639s" totalImagesPullingTime="4.931177048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:38.493251273 +0000 UTC m=+801.295337793" watchObservedRunningTime="2026-07-08 17:46:38.493902639 +0000 UTC m=+801.295989157" Jul 08 17:46:44.514190 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:44.514152 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-b4hpw" Jul 08 17:46:45.437038 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.437000 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b5db48b5c-7h6m2"] Jul 08 17:46:45.443904 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.443865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:45.447786 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.447757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-djkd2\"" Jul 08 17:46:45.457364 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.457326 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b5db48b5c-7h6m2"] Jul 08 17:46:45.524091 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.524053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84n5\" (UniqueName: \"kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5\") pod \"authorino-5b5db48b5c-7h6m2\" (UID: \"6eaabf48-da5a-4047-a805-d8ad91d70699\") " pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:45.551922 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.551880 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b5db48b5c-7h6m2"] Jul 08 17:46:45.552173 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:46:45.552154 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p84n5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" podUID="6eaabf48-da5a-4047-a805-d8ad91d70699" Jul 08 17:46:45.625407 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.625360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84n5\" (UniqueName: \"kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5\") pod \"authorino-5b5db48b5c-7h6m2\" (UID: \"6eaabf48-da5a-4047-a805-d8ad91d70699\") " pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:45.634772 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:45.634737 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84n5\" (UniqueName: \"kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5\") pod \"authorino-5b5db48b5c-7h6m2\" (UID: \"6eaabf48-da5a-4047-a805-d8ad91d70699\") " pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:46.516652 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:46.516614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:46.524331 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:46.524303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:46.532786 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:46.532755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p84n5\" (UniqueName: \"kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5\") pod \"6eaabf48-da5a-4047-a805-d8ad91d70699\" (UID: \"6eaabf48-da5a-4047-a805-d8ad91d70699\") " Jul 08 17:46:46.535108 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:46.535078 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5" (OuterVolumeSpecName: "kube-api-access-p84n5") pod "6eaabf48-da5a-4047-a805-d8ad91d70699" (UID: "6eaabf48-da5a-4047-a805-d8ad91d70699"). InnerVolumeSpecName "kube-api-access-p84n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:46.633794 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:46.633756 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84n5\" (UniqueName: \"kubernetes.io/projected/6eaabf48-da5a-4047-a805-d8ad91d70699-kube-api-access-p84n5\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:46:47.521260 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:47.521226 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b5db48b5c-7h6m2" Jul 08 17:46:47.559031 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:47.558996 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b5db48b5c-7h6m2"] Jul 08 17:46:47.562217 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:47.562186 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b5db48b5c-7h6m2"] Jul 08 17:46:47.792901 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:46:47.792818 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaabf48-da5a-4047-a805-d8ad91d70699" path="/var/lib/kubelet/pods/6eaabf48-da5a-4047-a805-d8ad91d70699/volumes" Jul 08 17:47:26.224298 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:26.224257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:47:43.771185 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:43.771148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:47:49.453225 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.453184 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq"] Jul 08 17:47:49.456868 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.456843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.460283 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.460261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 17:47:49.460421 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.460307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zg5mg\"" Jul 08 17:47:49.460421 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.460324 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 17:47:49.460421 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.460353 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 08 17:47:49.466213 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.466185 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq"] Jul 08 17:47:49.580105 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75wh\" (UniqueName: \"kubernetes.io/projected/a6bdd59b-06df-4203-b427-17b9d148f688-kube-api-access-q75wh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.580293 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.580293 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580151 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.580293 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6bdd59b-06df-4203-b427-17b9d148f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.580293 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.580293 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.580292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681508 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681716 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75wh\" (UniqueName: \"kubernetes.io/projected/a6bdd59b-06df-4203-b427-17b9d148f688-kube-api-access-q75wh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681716 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681716 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681716 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6bdd59b-06df-4203-b427-17b9d148f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.681716 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.681649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.682043 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.682021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.682102 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.682087 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.682141 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.682117 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.683944 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.683923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6bdd59b-06df-4203-b427-17b9d148f688-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.684087 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.684070 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6bdd59b-06df-4203-b427-17b9d148f688-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.690367 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.690343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75wh\" (UniqueName: \"kubernetes.io/projected/a6bdd59b-06df-4203-b427-17b9d148f688-kube-api-access-q75wh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sqrjq\" (UID: \"a6bdd59b-06df-4203-b427-17b9d148f688\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.767589 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.767510 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:49.913549 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:49.913522 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq"] Jul 08 17:47:49.915453 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:47:49.915427 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bdd59b_06df_4203_b427_17b9d148f688.slice/crio-31a87ea4c9a16bff20dbad7816562255ffa6539be3e788db185a07f75d03c452 WatchSource:0}: Error finding container 31a87ea4c9a16bff20dbad7816562255ffa6539be3e788db185a07f75d03c452: Status 404 returned error can't find the container with id 31a87ea4c9a16bff20dbad7816562255ffa6539be3e788db185a07f75d03c452 Jul 08 17:47:50.259644 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:50.259561 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:47:50.828241 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:50.828200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" event={"ID":"a6bdd59b-06df-4203-b427-17b9d148f688","Type":"ContainerStarted","Data":"31a87ea4c9a16bff20dbad7816562255ffa6539be3e788db185a07f75d03c452"} Jul 08 17:47:51.834389 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:51.834351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" event={"ID":"a6bdd59b-06df-4203-b427-17b9d148f688","Type":"ContainerStarted","Data":"2c819866e57fb95363a9e3bc680998fb077cea80cff031eb42f6b1734287a088"} Jul 08 17:47:51.857819 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:51.857772 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" podStartSLOduration=1.797454704 podStartE2EDuration="2.857757155s" totalImagesPullingTime="1.060302451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:51.855848982 +0000 UTC m=+874.657935501" watchObservedRunningTime="2026-07-08 17:47:51.857757155 +0000 UTC m=+874.659843674" Jul 08 17:47:58.069983 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:58.069944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:47:59.768692 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:59.768656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:59.769072 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:59.768721 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:59.781212 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:59.781184 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:47:59.885161 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:47:59.885134 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sqrjq" Jul 08 17:48:04.463166 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:04.463134 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:48:17.567297 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.567218 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw"] Jul 08 17:48:17.571076 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.571055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.573444 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.573423 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 17:48:17.585424 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.585403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw"] Jul 08 17:48:17.695626 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.695577 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:48:17.699737 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.699714 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:48:17.722966 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.722942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e550abd-0e78-4483-8584-85b2280c5fd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.723081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.722979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.723081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.723024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.723081 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.723060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.723249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.723114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24vw\" (UniqueName: \"kubernetes.io/projected/7e550abd-0e78-4483-8584-85b2280c5fd4-kube-api-access-m24vw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.723249 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.723195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823717 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823717 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823717 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24vw\" (UniqueName: \"kubernetes.io/projected/7e550abd-0e78-4483-8584-85b2280c5fd4-kube-api-access-m24vw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823976 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823976 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e550abd-0e78-4483-8584-85b2280c5fd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.823976 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.823849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.824134 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.824111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.824210 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.824193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.824279 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.824267 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.825897 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.825880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e550abd-0e78-4483-8584-85b2280c5fd4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.826650 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.826634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 17:48:17.834102 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.834085 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24vw\" (UniqueName: \"kubernetes.io/projected/7e550abd-0e78-4483-8584-85b2280c5fd4-kube-api-access-m24vw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.836114 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.836093 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e550abd-0e78-4483-8584-85b2280c5fd4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ncpxw\" (UID: \"7e550abd-0e78-4483-8584-85b2280c5fd4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:17.909159 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:17.909125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:18.049389 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:18.049363 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw"] Jul 08 17:48:18.051091 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:48:18.051065 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e550abd_0e78_4483_8584_85b2280c5fd4.slice/crio-37fcc116678cfed282f536adc53d04a1171a455d8214cceb49dae490cbc2577d WatchSource:0}: Error finding container 37fcc116678cfed282f536adc53d04a1171a455d8214cceb49dae490cbc2577d: Status 404 returned error can't find the container with id 37fcc116678cfed282f536adc53d04a1171a455d8214cceb49dae490cbc2577d Jul 08 17:48:18.052447 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:18.052431 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:48:18.964325 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:18.964289 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" event={"ID":"7e550abd-0e78-4483-8584-85b2280c5fd4","Type":"ContainerStarted","Data":"573ac602263c64214c43b546c139a4a9707e3e998f95d80230fd2e25886dcd0e"} Jul 08 17:48:18.964856 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:18.964348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" event={"ID":"7e550abd-0e78-4483-8584-85b2280c5fd4","Type":"ContainerStarted","Data":"37fcc116678cfed282f536adc53d04a1171a455d8214cceb49dae490cbc2577d"} Jul 08 17:48:18.982577 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:18.982523 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" podStartSLOduration=1.7635124370000002 podStartE2EDuration="1.982503838s" totalImagesPullingTime="218.991401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:48:18.98185686 +0000 UTC m=+901.783943379" watchObservedRunningTime="2026-07-08 17:48:18.982503838 +0000 UTC m=+901.784590358" Jul 08 17:48:19.565678 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:19.565644 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:48:27.909484 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:27.909443 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:27.909932 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:27.909506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:27.928191 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:27.928155 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:28.023205 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:28.023177 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ncpxw" Jul 08 17:48:31.459415 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:48:31.459377 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:49:45.775709 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:49:45.775615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:49:54.961610 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:49:54.961549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:50:07.032253 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:50:07.032220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:50:16.461003 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:50:16.460969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:50:25.752554 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:50:25.752508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:51:32.354028 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:51:32.353941 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:51:42.257812 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:51:42.257778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:51:50.656158 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:51:50.656121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:51:59.664859 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:51:59.664823 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:52:10.068315 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:52:10.068279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:52:19.662166 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:52:19.662131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:52:31.358924 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:52:31.358895 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:52:41.060305 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:52:41.060219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:53:03.955319 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:03.955280 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:53:15.960803 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:15.960767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:53:17.726567 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:17.726537 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:53:17.730909 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:17.730890 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:53:39.355954 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:39.355921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:53:49.751751 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:53:49.751710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:54:05.271135 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:54:05.271098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:54:15.571967 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:54:15.571871 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:54:25.355983 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:54:25.355942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:54:34.961163 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:54:34.961130 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:54:44.360131 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:54:44.360087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:55:10.473340 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:10.473302 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:55:10.475893 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:10.473516 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" podUID="5283e5bd-0c59-4173-92b9-bc433cd708d5" containerName="manager" containerID="cri-o://153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265" gracePeriod=10 Jul 08 17:55:13.938824 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:13.938800 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:55:14.004905 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.004864 2579 generic.go:356] "Generic (PLEG): container finished" podID="5283e5bd-0c59-4173-92b9-bc433cd708d5" containerID="153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265" exitCode=0 Jul 08 17:55:14.005092 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.004908 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" Jul 08 17:55:14.005092 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.004932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" event={"ID":"5283e5bd-0c59-4173-92b9-bc433cd708d5","Type":"ContainerDied","Data":"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265"} Jul 08 17:55:14.005092 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.004984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp" event={"ID":"5283e5bd-0c59-4173-92b9-bc433cd708d5","Type":"ContainerDied","Data":"1b58ecf3589de6857db617f458bd2716f7b3fff5e5267dd0e9b7ced82b9c39d7"} Jul 08 17:55:14.005092 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.005005 2579 scope.go:117] "RemoveContainer" containerID="153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265" Jul 08 17:55:14.014799 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.014773 2579 scope.go:117] "RemoveContainer" containerID="153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265" Jul 08 17:55:14.015065 ip-10-0-129-65 kubenswrapper[2579]: E0708 17:55:14.015047 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265\": container with ID starting with 153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265 not found: ID does not exist" containerID="153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265" Jul 08 17:55:14.015134 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.015076 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265"} err="failed to get container status \"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265\": rpc error: code = NotFound desc = could not find container \"153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265\": container with ID starting with 153cba975898437c4f3cd92523f88f35c188c9903b309f5a937a720b61466265 not found: ID does not exist" Jul 08 17:55:14.027409 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.027385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glvpr\" (UniqueName: \"kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr\") pod \"5283e5bd-0c59-4173-92b9-bc433cd708d5\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " Jul 08 17:55:14.027518 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.027475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume\") pod \"5283e5bd-0c59-4173-92b9-bc433cd708d5\" (UID: \"5283e5bd-0c59-4173-92b9-bc433cd708d5\") " Jul 08 17:55:14.027872 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.027853 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5283e5bd-0c59-4173-92b9-bc433cd708d5" (UID: "5283e5bd-0c59-4173-92b9-bc433cd708d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:55:14.029569 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.029544 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr" (OuterVolumeSpecName: "kube-api-access-glvpr") pod "5283e5bd-0c59-4173-92b9-bc433cd708d5" (UID: "5283e5bd-0c59-4173-92b9-bc433cd708d5"). InnerVolumeSpecName "kube-api-access-glvpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:55:14.128656 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.128550 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glvpr\" (UniqueName: \"kubernetes.io/projected/5283e5bd-0c59-4173-92b9-bc433cd708d5-kube-api-access-glvpr\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:55:14.128656 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.128590 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5283e5bd-0c59-4173-92b9-bc433cd708d5-extensions-socket-volume\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 17:55:14.325999 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.325963 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:55:14.332146 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:14.332115 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-96gsp"] Jul 08 17:55:15.791714 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:55:15.791678 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5283e5bd-0c59-4173-92b9-bc433cd708d5" path="/var/lib/kubelet/pods/5283e5bd-0c59-4173-92b9-bc433cd708d5/volumes" Jul 08 17:56:11.702534 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.702493 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq"] Jul 08 17:56:11.703058 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.702893 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5283e5bd-0c59-4173-92b9-bc433cd708d5" containerName="manager" Jul 08 17:56:11.703058 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.702904 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5283e5bd-0c59-4173-92b9-bc433cd708d5" containerName="manager" Jul 08 17:56:11.703058 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.702980 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5283e5bd-0c59-4173-92b9-bc433cd708d5" containerName="manager" Jul 08 17:56:11.706069 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.706047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.709413 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.709394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-799hk\"" Jul 08 17:56:11.723690 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.723663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq"] Jul 08 17:56:11.741575 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.741539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwbp8\" (UniqueName: \"kubernetes.io/projected/85b1610a-4937-4054-9026-c97b70bc9a55-kube-api-access-nwbp8\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.741734 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.741578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b1610a-4937-4054-9026-c97b70bc9a55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.842843 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.842797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b1610a-4937-4054-9026-c97b70bc9a55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.843019 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.842914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwbp8\" (UniqueName: \"kubernetes.io/projected/85b1610a-4937-4054-9026-c97b70bc9a55-kube-api-access-nwbp8\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.843237 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.843215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b1610a-4937-4054-9026-c97b70bc9a55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:11.872985 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:11.872949 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwbp8\" (UniqueName: \"kubernetes.io/projected/85b1610a-4937-4054-9026-c97b70bc9a55-kube-api-access-nwbp8\") pod \"kuadrant-operator-controller-manager-86878c769f-kdcvq\" (UID: \"85b1610a-4937-4054-9026-c97b70bc9a55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:12.016547 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.016453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:12.179382 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.179350 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq"] Jul 08 17:56:12.182448 ip-10-0-129-65 kubenswrapper[2579]: W0708 17:56:12.182422 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b1610a_4937_4054_9026_c97b70bc9a55.slice/crio-24445b90f45a108a4154bd792ad58909bae4134ce93331638b3a125102f18513 WatchSource:0}: Error finding container 24445b90f45a108a4154bd792ad58909bae4134ce93331638b3a125102f18513: Status 404 returned error can't find the container with id 24445b90f45a108a4154bd792ad58909bae4134ce93331638b3a125102f18513 Jul 08 17:56:12.292813 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.292718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" event={"ID":"85b1610a-4937-4054-9026-c97b70bc9a55","Type":"ContainerStarted","Data":"94a5fb98c67a11ad8bde215637fe2c61e4135e1978ed9b8a3f99e4adc2e06dbe"} Jul 08 17:56:12.292813 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.292778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" event={"ID":"85b1610a-4937-4054-9026-c97b70bc9a55","Type":"ContainerStarted","Data":"24445b90f45a108a4154bd792ad58909bae4134ce93331638b3a125102f18513"} Jul 08 17:56:12.292813 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.292806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:12.313701 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:12.313629 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" podStartSLOduration=1.313586055 podStartE2EDuration="1.313586055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:56:12.311570767 +0000 UTC m=+1375.113657289" watchObservedRunningTime="2026-07-08 17:56:12.313586055 +0000 UTC m=+1375.115672576" Jul 08 17:56:23.299869 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:23.299837 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-kdcvq" Jul 08 17:56:32.861172 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:32.861129 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:56:44.754000 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:44.753964 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:56:54.967670 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:56:54.967635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:06.458430 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:06.458395 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:17.879188 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:17.879114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:28.179157 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:28.179115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:36.371730 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:36.371693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:47.374143 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:47.374102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:57:59.970862 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:57:59.970826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:58:07.569786 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:07.569756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:58:17.759536 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:17.759504 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:58:17.763787 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:17.763752 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 17:58:19.876306 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:19.876259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:58:29.265055 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:29.265015 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:58:49.968340 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:49.968255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:58:57.458691 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:58:57.458650 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:07.193694 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:07.193663 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:17.084063 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:17.084034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:26.672294 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:26.672257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:38.522743 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:38.522701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:48.454998 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:48.454965 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 17:59:57.161247 ip-10-0-129-65 kubenswrapper[2579]: I0708 17:59:57.161205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:00:08.467134 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:00:08.467091 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:00:17.973703 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:00:17.973670 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:00:27.794142 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:00:27.794110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:00:41.399460 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:00:41.399419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:00:50.961874 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:00:50.961831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:01:05.071251 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:01:05.071217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:01:17.691537 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:01:17.691500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:01:25.781220 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:01:25.781179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:01:38.179715 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:01:38.179687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:01:54.772072 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:01:54.772031 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:02:06.267636 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:02:06.264483 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:02:17.883322 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:02:17.883286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:02:28.465611 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:02:28.465561 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:02:42.163125 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:02:42.163093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:02:50.868542 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:02:50.868505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:03:00.760502 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:00.760466 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:03:09.552306 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:09.552266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:03:17.788650 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:17.788615 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:03:17.795531 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:17.795501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:03:31.958769 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:31.958735 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:03:40.966892 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:03:40.966855 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:05:03.178790 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:05:03.178745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:05:10.593715 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:05:10.593671 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:05:19.364165 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:05:19.364129 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:05:29.270268 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:05:29.270228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:05:35.856219 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:05:35.856188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:06:09.655754 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:06:09.655676 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:06:23.310905 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:06:23.310874 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:06:39.162652 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:06:39.162611 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:06:49.857934 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:06:49.857904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:06:59.389886 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:06:59.389849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:07:34.470477 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:07:34.470435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:07:43.314752 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:07:43.314667 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:07:55.360590 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:07:55.360550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:08:17.818707 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:08:17.818677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:08:17.825461 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:08:17.825435 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:08:48.169051 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:08:48.169019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:08:59.758117 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:08:59.758085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:09:09.067240 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:09:09.067155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:09:22.461071 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:09:22.461033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:09:39.093317 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:09:39.093280 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:10:03.261451 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:10:03.261419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:10:45.162731 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:10:45.162652 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:10:58.269261 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:10:58.269231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:11:11.262203 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:11:11.262164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:11:28.064456 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:11:28.064420 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:11:39.557048 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:11:39.557007 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:11:51.672631 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:11:51.672574 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:12:13.888238 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:12:13.888163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:12:32.474905 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:12:32.474870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:12:55.274403 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:12:55.274371 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:13:12.963014 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:13:12.962973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:13:17.848151 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:13:17.848123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:13:17.855691 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:13:17.855665 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:13:31.562532 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:13:31.562499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:13:48.059096 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:13:48.059023 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:14:04.165937 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:14:04.165904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:14:21.262357 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:14:21.262324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:14:37.263058 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:14:37.263024 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:14:56.162950 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:14:56.162918 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:15:00.192883 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.192833 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:15:00.196549 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.196527 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:15:00.196702 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.196667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:15:00.199754 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.199732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:15:00.201376 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.201358 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fljv6\"" Jul 08 18:15:00.225745 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.225721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:15:00.229410 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.229387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:15:00.295156 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.295121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5p49\" (UniqueName: \"kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49\") pod \"maas-api-key-cleanup-29725575-5zmsv\" (UID: \"ebcaebab-5469-4ea9-a9ed-24fd8fbba610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:15:00.295307 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.295184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzv2f\" (UniqueName: \"kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f\") pod \"maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4\" (UID: \"dd2a4bfa-1812-4623-81dc-07fc93470d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:15:00.396152 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.396125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5p49\" (UniqueName: \"kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49\") pod \"maas-api-key-cleanup-29725575-5zmsv\" (UID: \"ebcaebab-5469-4ea9-a9ed-24fd8fbba610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:15:00.396294 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.396168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzv2f\" (UniqueName: \"kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f\") pod \"maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4\" (UID: \"dd2a4bfa-1812-4623-81dc-07fc93470d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:15:00.405370 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.405341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzv2f\" (UniqueName: \"kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f\") pod \"maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4\" (UID: \"dd2a4bfa-1812-4623-81dc-07fc93470d0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:15:00.405477 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.405384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5p49\" (UniqueName: \"kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49\") pod \"maas-api-key-cleanup-29725575-5zmsv\" (UID: \"ebcaebab-5469-4ea9-a9ed-24fd8fbba610\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:15:00.509555 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.509483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:15:00.516467 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.516447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:15:00.661084 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.661050 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:15:00.661809 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:15:00.661784 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2a4bfa_1812_4623_81dc_07fc93470d0f.slice/crio-60a4eb24192948cadd57b29a2a92946688e4d32af81f4a000224bdd77566e72f WatchSource:0}: Error finding container 60a4eb24192948cadd57b29a2a92946688e4d32af81f4a000224bdd77566e72f: Status 404 returned error can't find the container with id 60a4eb24192948cadd57b29a2a92946688e4d32af81f4a000224bdd77566e72f Jul 08 18:15:00.664571 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.664218 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:15:00.703119 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.703093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:15:00.709468 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:15:00.709442 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcaebab_5469_4ea9_a9ed_24fd8fbba610.slice/crio-c64684d79eced695b1bac0db292801f39860717596b96357b709af3b31e5d92b WatchSource:0}: Error finding container c64684d79eced695b1bac0db292801f39860717596b96357b709af3b31e5d92b: Status 404 returned error can't find the container with id c64684d79eced695b1bac0db292801f39860717596b96357b709af3b31e5d92b Jul 08 18:15:00.825423 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.825333 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" event={"ID":"dd2a4bfa-1812-4623-81dc-07fc93470d0f","Type":"ContainerStarted","Data":"60a4eb24192948cadd57b29a2a92946688e4d32af81f4a000224bdd77566e72f"} Jul 08 18:15:00.826562 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:00.826539 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" event={"ID":"ebcaebab-5469-4ea9-a9ed-24fd8fbba610","Type":"ContainerStarted","Data":"c64684d79eced695b1bac0db292801f39860717596b96357b709af3b31e5d92b"} Jul 08 18:15:03.842975 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:03.842937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" event={"ID":"ebcaebab-5469-4ea9-a9ed-24fd8fbba610","Type":"ContainerStarted","Data":"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546"} Jul 08 18:15:03.859121 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:03.859068 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" podStartSLOduration=1.324811035 podStartE2EDuration="3.859051266s" totalImagesPullingTime="2.534240231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:03.858121731 +0000 UTC m=+2506.660208240" watchObservedRunningTime="2026-07-08 18:15:03.859051266 +0000 UTC m=+2506.661137785" Jul 08 18:15:05.855405 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:05.855370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" event={"ID":"dd2a4bfa-1812-4623-81dc-07fc93470d0f","Type":"ContainerStarted","Data":"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37"} Jul 08 18:15:05.875028 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:05.874980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" podStartSLOduration=1.970464141 podStartE2EDuration="5.874964445s" totalImagesPullingTime="3.904500304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:05.874292683 +0000 UTC m=+2508.676379204" watchObservedRunningTime="2026-07-08 18:15:05.874964445 +0000 UTC m=+2508.677050999" Jul 08 18:15:21.155081 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:21.154998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:15:35.665653 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:35.665621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:15:55.259253 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:15:55.259219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:16:00.019677 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:00.019640 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:16:00.020232 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:00.019904 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" podUID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" containerName="cleanup" containerID="cri-o://353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37" gracePeriod=30 Jul 08 18:16:00.021451 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:00.021426 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:16:00.021629 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:00.021591 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" podUID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" containerName="cleanup" containerID="cri-o://9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546" gracePeriod=30 Jul 08 18:16:14.762900 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:14.762861 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:16:27.657280 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:27.657238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:16:30.225382 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.225353 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:16:30.241958 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.241925 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:16:30.291983 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.291902 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" containerID="9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546" exitCode=137 Jul 08 18:16:30.291983 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.291941 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" Jul 08 18:16:30.292173 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.291980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" event={"ID":"ebcaebab-5469-4ea9-a9ed-24fd8fbba610","Type":"ContainerDied","Data":"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546"} Jul 08 18:16:30.292173 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.292031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv" event={"ID":"ebcaebab-5469-4ea9-a9ed-24fd8fbba610","Type":"ContainerDied","Data":"c64684d79eced695b1bac0db292801f39860717596b96357b709af3b31e5d92b"} Jul 08 18:16:30.292173 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.292059 2579 scope.go:117] "RemoveContainer" containerID="9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546" Jul 08 18:16:30.293305 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.293285 2579 generic.go:356] "Generic (PLEG): container finished" podID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" containerID="353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37" exitCode=137 Jul 08 18:16:30.293390 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.293328 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" Jul 08 18:16:30.293390 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.293368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" event={"ID":"dd2a4bfa-1812-4623-81dc-07fc93470d0f","Type":"ContainerDied","Data":"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37"} Jul 08 18:16:30.293480 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.293413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4" event={"ID":"dd2a4bfa-1812-4623-81dc-07fc93470d0f","Type":"ContainerDied","Data":"60a4eb24192948cadd57b29a2a92946688e4d32af81f4a000224bdd77566e72f"} Jul 08 18:16:30.296975 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.296956 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5p49\" (UniqueName: \"kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49\") pod \"ebcaebab-5469-4ea9-a9ed-24fd8fbba610\" (UID: \"ebcaebab-5469-4ea9-a9ed-24fd8fbba610\") " Jul 08 18:16:30.297094 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.297017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzv2f\" (UniqueName: \"kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f\") pod \"dd2a4bfa-1812-4623-81dc-07fc93470d0f\" (UID: \"dd2a4bfa-1812-4623-81dc-07fc93470d0f\") " Jul 08 18:16:30.304590 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.304498 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49" (OuterVolumeSpecName: "kube-api-access-k5p49") pod "ebcaebab-5469-4ea9-a9ed-24fd8fbba610" (UID: "ebcaebab-5469-4ea9-a9ed-24fd8fbba610"). InnerVolumeSpecName "kube-api-access-k5p49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:30.304590 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.304514 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f" (OuterVolumeSpecName: "kube-api-access-mzv2f") pod "dd2a4bfa-1812-4623-81dc-07fc93470d0f" (UID: "dd2a4bfa-1812-4623-81dc-07fc93470d0f"). InnerVolumeSpecName "kube-api-access-mzv2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:30.305515 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.305491 2579 scope.go:117] "RemoveContainer" containerID="9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546" Jul 08 18:16:30.305814 ip-10-0-129-65 kubenswrapper[2579]: E0708 18:16:30.305789 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546\": container with ID starting with 9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546 not found: ID does not exist" containerID="9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546" Jul 08 18:16:30.305888 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.305826 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546"} err="failed to get container status \"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546\": rpc error: code = NotFound desc = could not find container \"9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546\": container with ID starting with 9c6a7aa6272473091f74cb50830a4e4dd4cbdb21913fc58aad8e6fa701203546 not found: ID does not exist" Jul 08 18:16:30.305888 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.305848 2579 scope.go:117] "RemoveContainer" containerID="353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37" Jul 08 18:16:30.313782 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.313765 2579 scope.go:117] "RemoveContainer" containerID="353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37" Jul 08 18:16:30.314024 ip-10-0-129-65 kubenswrapper[2579]: E0708 18:16:30.314007 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37\": container with ID starting with 353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37 not found: ID does not exist" containerID="353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37" Jul 08 18:16:30.314085 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.314042 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37"} err="failed to get container status \"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37\": rpc error: code = NotFound desc = could not find container \"353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37\": container with ID starting with 353ded29f98b16ca47b9b31a3069ade5a62fac5ab771d5e81e52c1207f237f37 not found: ID does not exist" Jul 08 18:16:30.398401 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.398375 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5p49\" (UniqueName: \"kubernetes.io/projected/ebcaebab-5469-4ea9-a9ed-24fd8fbba610-kube-api-access-k5p49\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 18:16:30.398401 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.398401 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzv2f\" (UniqueName: \"kubernetes.io/projected/dd2a4bfa-1812-4623-81dc-07fc93470d0f-kube-api-access-mzv2f\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 18:16:30.622128 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.622094 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:16:30.624051 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.624028 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-5zmsv"] Jul 08 18:16:30.646584 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.646557 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:16:30.650905 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:30.650885 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-608a65-29725575-k2fd4"] Jul 08 18:16:31.792103 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:31.792073 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" path="/var/lib/kubelet/pods/dd2a4bfa-1812-4623-81dc-07fc93470d0f/volumes" Jul 08 18:16:31.792522 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:31.792373 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" path="/var/lib/kubelet/pods/ebcaebab-5469-4ea9-a9ed-24fd8fbba610/volumes" Jul 08 18:16:45.764663 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:16:45.764630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:17:00.856588 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:00.856550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:17:18.781350 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:18.781314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:17:27.428284 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428247 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029662000"] Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428638 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" containerName="cleanup" Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428649 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" containerName="cleanup" Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428673 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" containerName="cleanup" Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428678 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" containerName="cleanup" Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428742 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcaebab-5469-4ea9-a9ed-24fd8fbba610" containerName="cleanup" Jul 08 18:17:27.428766 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.428751 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2a4bfa-1812-4623-81dc-07fc93470d0f" containerName="cleanup" Jul 08 18:17:27.431441 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.431424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:27.437348 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.437323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029662000"] Jul 08 18:17:27.487294 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.487266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7h2\" (UniqueName: \"kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2\") pod \"test-curl-7634-140482029662000\" (UID: \"ede1bb8f-3046-4279-bde0-578b90a87aa6\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:27.587983 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.587944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7h2\" (UniqueName: \"kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2\") pod \"test-curl-7634-140482029662000\" (UID: \"ede1bb8f-3046-4279-bde0-578b90a87aa6\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:27.597402 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.597366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7h2\" (UniqueName: \"kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2\") pod \"test-curl-7634-140482029662000\" (UID: \"ede1bb8f-3046-4279-bde0-578b90a87aa6\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:27.741342 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.741298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:27.881365 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:27.881338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029662000"] Jul 08 18:17:27.883244 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:17:27.883208 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede1bb8f_3046_4279_bde0_578b90a87aa6.slice/crio-b4fdfb9ca3906a8f7fb9269212f878fc010789c535984f9540d4e06e9bf773c1 WatchSource:0}: Error finding container b4fdfb9ca3906a8f7fb9269212f878fc010789c535984f9540d4e06e9bf773c1: Status 404 returned error can't find the container with id b4fdfb9ca3906a8f7fb9269212f878fc010789c535984f9540d4e06e9bf773c1 Jul 08 18:17:28.592246 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:28.592212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" event={"ID":"ede1bb8f-3046-4279-bde0-578b90a87aa6","Type":"ContainerStarted","Data":"b4fdfb9ca3906a8f7fb9269212f878fc010789c535984f9540d4e06e9bf773c1"} Jul 08 18:17:29.599923 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:29.599880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" event={"ID":"ede1bb8f-3046-4279-bde0-578b90a87aa6","Type":"ContainerStarted","Data":"195ad63eff2bdfaae7d4e74676f966f6392c56ed69ff003506f95ee47cfeb1d2"} Jul 08 18:17:29.615580 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:29.615530 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" podStartSLOduration=1.914520616 podStartE2EDuration="2.61551384s" totalImagesPullingTime="700.993224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:17:29.614508072 +0000 UTC m=+2652.416594592" watchObservedRunningTime="2026-07-08 18:17:29.61551384 +0000 UTC m=+2652.417600359" Jul 08 18:17:37.602498 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:37.602458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:17:39.650493 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:39.650458 2579 generic.go:356] "Generic (PLEG): container finished" podID="ede1bb8f-3046-4279-bde0-578b90a87aa6" containerID="195ad63eff2bdfaae7d4e74676f966f6392c56ed69ff003506f95ee47cfeb1d2" exitCode=28 Jul 08 18:17:39.650890 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:39.650536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" event={"ID":"ede1bb8f-3046-4279-bde0-578b90a87aa6","Type":"ContainerDied","Data":"195ad63eff2bdfaae7d4e74676f966f6392c56ed69ff003506f95ee47cfeb1d2"} Jul 08 18:17:40.810921 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:40.810892 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:40.845378 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:40.845349 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029662000"] Jul 08 18:17:40.849399 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:40.849375 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029662000"] Jul 08 18:17:40.911371 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:40.911294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl7h2\" (UniqueName: \"kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2\") pod \"ede1bb8f-3046-4279-bde0-578b90a87aa6\" (UID: \"ede1bb8f-3046-4279-bde0-578b90a87aa6\") " Jul 08 18:17:40.913571 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:40.913548 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2" (OuterVolumeSpecName: "kube-api-access-nl7h2") pod "ede1bb8f-3046-4279-bde0-578b90a87aa6" (UID: "ede1bb8f-3046-4279-bde0-578b90a87aa6"). InnerVolumeSpecName "kube-api-access-nl7h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:41.012808 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.012775 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7h2\" (UniqueName: \"kubernetes.io/projected/ede1bb8f-3046-4279-bde0-578b90a87aa6-kube-api-access-nl7h2\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.021399 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.021366 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029660592"] Jul 08 18:17:41.022613 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.022572 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede1bb8f-3046-4279-bde0-578b90a87aa6" containerName="test-curl-7634-140482029662000" Jul 08 18:17:41.023224 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.023183 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede1bb8f-3046-4279-bde0-578b90a87aa6" containerName="test-curl-7634-140482029662000" Jul 08 18:17:41.023813 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.023790 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede1bb8f-3046-4279-bde0-578b90a87aa6" containerName="test-curl-7634-140482029662000" Jul 08 18:17:41.027637 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.027615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029660592"] Jul 08 18:17:41.027737 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.027700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:41.114244 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.114206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xk2v\" (UniqueName: \"kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v\") pod \"test-curl-7634-140482029660592\" (UID: \"df21cacc-e6b1-4c21-8c73-706c4fa945d1\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:41.215060 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.215032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xk2v\" (UniqueName: \"kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v\") pod \"test-curl-7634-140482029660592\" (UID: \"df21cacc-e6b1-4c21-8c73-706c4fa945d1\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:41.224419 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.224395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xk2v\" (UniqueName: \"kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v\") pod \"test-curl-7634-140482029660592\" (UID: \"df21cacc-e6b1-4c21-8c73-706c4fa945d1\") " pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:41.339337 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.339305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:41.480317 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.480291 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029660592"] Jul 08 18:17:41.482462 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:17:41.482435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf21cacc_e6b1_4c21_8c73_706c4fa945d1.slice/crio-de8d8e998cd5985b5c0554a3ac7119f723797b08a50cf26e3579aeebcbb9a672 WatchSource:0}: Error finding container de8d8e998cd5985b5c0554a3ac7119f723797b08a50cf26e3579aeebcbb9a672: Status 404 returned error can't find the container with id de8d8e998cd5985b5c0554a3ac7119f723797b08a50cf26e3579aeebcbb9a672 Jul 08 18:17:41.662055 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.662019 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4fdfb9ca3906a8f7fb9269212f878fc010789c535984f9540d4e06e9bf773c1" Jul 08 18:17:41.662055 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.662040 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:41.663253 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.663223 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" event={"ID":"df21cacc-e6b1-4c21-8c73-706c4fa945d1","Type":"ContainerStarted","Data":"de8d8e998cd5985b5c0554a3ac7119f723797b08a50cf26e3579aeebcbb9a672"} Jul 08 18:17:41.672794 ip-10-0-129-65 kubenswrapper[2579]: E0708 18:17:41.672764 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7634-140482029662000\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="ede1bb8f-3046-4279-bde0-578b90a87aa6" pod="odh-ai-gateway-infra/test-curl-7634-140482029662000" Jul 08 18:17:41.795360 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:41.795325 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede1bb8f-3046-4279-bde0-578b90a87aa6" path="/var/lib/kubelet/pods/ede1bb8f-3046-4279-bde0-578b90a87aa6/volumes" Jul 08 18:17:42.670165 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:42.670132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" event={"ID":"df21cacc-e6b1-4c21-8c73-706c4fa945d1","Type":"ContainerStarted","Data":"e6fbb1dfd55d7452a3650749da682d7a15025c7aff9ab44a119ce143b9db26f3"} Jul 08 18:17:42.687930 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:42.687885 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" podStartSLOduration=1.38796472 podStartE2EDuration="1.687869999s" totalImagesPullingTime="299.905279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:17:42.687591473 +0000 UTC m=+2665.489678004" watchObservedRunningTime="2026-07-08 18:17:42.687869999 +0000 UTC m=+2665.489956518" Jul 08 18:17:52.720647 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:52.720586 2579 generic.go:356] "Generic (PLEG): container finished" podID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" containerID="e6fbb1dfd55d7452a3650749da682d7a15025c7aff9ab44a119ce143b9db26f3" exitCode=28 Jul 08 18:17:52.721024 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:52.720655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" event={"ID":"df21cacc-e6b1-4c21-8c73-706c4fa945d1","Type":"ContainerDied","Data":"e6fbb1dfd55d7452a3650749da682d7a15025c7aff9ab44a119ce143b9db26f3"} Jul 08 18:17:53.860744 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:53.860719 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:53.891169 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:53.891125 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029660592"] Jul 08 18:17:53.897925 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:53.897896 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482029660592"] Jul 08 18:17:54.025126 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.025038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xk2v\" (UniqueName: \"kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v\") pod \"df21cacc-e6b1-4c21-8c73-706c4fa945d1\" (UID: \"df21cacc-e6b1-4c21-8c73-706c4fa945d1\") " Jul 08 18:17:54.027306 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.027283 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v" (OuterVolumeSpecName: "kube-api-access-7xk2v") pod "df21cacc-e6b1-4c21-8c73-706c4fa945d1" (UID: "df21cacc-e6b1-4c21-8c73-706c4fa945d1"). InnerVolumeSpecName "kube-api-access-7xk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:54.086282 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.086244 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482028745040"] Jul 08 18:17:54.086648 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.086635 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" containerName="test-curl-7634-140482029660592" Jul 08 18:17:54.086688 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.086650 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" containerName="test-curl-7634-140482029660592" Jul 08 18:17:54.086738 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.086729 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" containerName="test-curl-7634-140482029660592" Jul 08 18:17:54.089751 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.089736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:17:54.094278 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.094251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482028745040"] Jul 08 18:17:54.126209 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.126178 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xk2v\" (UniqueName: \"kubernetes.io/projected/df21cacc-e6b1-4c21-8c73-706c4fa945d1-kube-api-access-7xk2v\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 18:17:54.227511 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.227480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4slp\" (UniqueName: \"kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp\") pod \"test-curl-7634-140482028745040\" (UID: \"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9\") " pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:17:54.328467 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.328377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4slp\" (UniqueName: \"kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp\") pod \"test-curl-7634-140482028745040\" (UID: \"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9\") " pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:17:54.337645 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.337585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4slp\" (UniqueName: \"kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp\") pod \"test-curl-7634-140482028745040\" (UID: \"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9\") " pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:17:54.399581 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.399543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:17:54.540618 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.540577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482028745040"] Jul 08 18:17:54.542693 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:17:54.542664 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9b18c31_cef4_464d_ae11_38bdcaa2a4e9.slice/crio-3973530fc4d890ff0c594b81a02a6ed6546863a2fa126859493e45a6688f4c98 WatchSource:0}: Error finding container 3973530fc4d890ff0c594b81a02a6ed6546863a2fa126859493e45a6688f4c98: Status 404 returned error can't find the container with id 3973530fc4d890ff0c594b81a02a6ed6546863a2fa126859493e45a6688f4c98 Jul 08 18:17:54.730996 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.730958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" event={"ID":"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9","Type":"ContainerStarted","Data":"3973530fc4d890ff0c594b81a02a6ed6546863a2fa126859493e45a6688f4c98"} Jul 08 18:17:54.732180 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.732158 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de8d8e998cd5985b5c0554a3ac7119f723797b08a50cf26e3579aeebcbb9a672" Jul 08 18:17:54.732332 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:54.732219 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:54.742532 ip-10-0-129-65 kubenswrapper[2579]: E0708 18:17:54.742504 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7634-140482029660592\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:55.738109 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:55.738072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" event={"ID":"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9","Type":"ContainerStarted","Data":"1754f1d788839057fff6763f5aa9e876b2e490c1f4d74c11e05d97e8e646bcc2"} Jul 08 18:17:55.753421 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:55.753376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" podStartSLOduration=1.4837428369999999 podStartE2EDuration="1.753361375s" totalImagesPullingTime="269.618538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:17:55.752436663 +0000 UTC m=+2678.554523183" watchObservedRunningTime="2026-07-08 18:17:55.753361375 +0000 UTC m=+2678.555447893" Jul 08 18:17:55.754155 ip-10-0-129-65 kubenswrapper[2579]: E0708 18:17:55.754124 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7634-140482029660592\" is forbidden: User \"system:node:ip-10-0-129-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-65.ec2.internal' and this object" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" pod="odh-ai-gateway-infra/test-curl-7634-140482029660592" Jul 08 18:17:55.791148 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:17:55.791119 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df21cacc-e6b1-4c21-8c73-706c4fa945d1" path="/var/lib/kubelet/pods/df21cacc-e6b1-4c21-8c73-706c4fa945d1/volumes" Jul 08 18:18:05.789418 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:05.789386 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" containerID="1754f1d788839057fff6763f5aa9e876b2e490c1f4d74c11e05d97e8e646bcc2" exitCode=28 Jul 08 18:18:05.795637 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:05.791845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" event={"ID":"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9","Type":"ContainerDied","Data":"1754f1d788839057fff6763f5aa9e876b2e490c1f4d74c11e05d97e8e646bcc2"} Jul 08 18:18:06.928970 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:06.928947 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:18:06.963271 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:06.963244 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482028745040"] Jul 08 18:18:06.967011 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:06.966985 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7634-140482028745040"] Jul 08 18:18:07.035817 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.035791 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4slp\" (UniqueName: \"kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp\") pod \"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9\" (UID: \"f9b18c31-cef4-464d-ae11-38bdcaa2a4e9\") " Jul 08 18:18:07.037724 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.037697 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp" (OuterVolumeSpecName: "kube-api-access-p4slp") pod "f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" (UID: "f9b18c31-cef4-464d-ae11-38bdcaa2a4e9"). InnerVolumeSpecName "kube-api-access-p4slp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:18:07.137099 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.137038 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4slp\" (UniqueName: \"kubernetes.io/projected/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9-kube-api-access-p4slp\") on node \"ip-10-0-129-65.ec2.internal\" DevicePath \"\"" Jul 08 18:18:07.791572 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.791541 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" path="/var/lib/kubelet/pods/f9b18c31-cef4-464d-ae11-38bdcaa2a4e9/volumes" Jul 08 18:18:07.799402 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.799378 2579 scope.go:117] "RemoveContainer" containerID="1754f1d788839057fff6763f5aa9e876b2e490c1f4d74c11e05d97e8e646bcc2" Jul 08 18:18:07.799509 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:07.799387 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7634-140482028745040" Jul 08 18:18:11.565438 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:11.565342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:18:17.876818 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:17.876789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:18:17.886340 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:17.886317 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:18:23.961822 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:23.961788 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-72t8w"] Jul 08 18:18:26.886136 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:26.886102 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fwl8z_16affe09-6490-406d-8019-d0e4c7c2bcf4/manager/0.log" Jul 08 18:18:27.009544 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:27.009513 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fjpln_8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0/manager/0.log" Jul 08 18:18:27.243391 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:27.243363 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x4f7b_194fd4db-b502-40ce-b463-9ef5d3339207/server/0.log" Jul 08 18:18:27.370422 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:27.370389 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r5txw_686114d0-cc38-45b7-89f5-ee98b6b42723/manager/0.log" Jul 08 18:18:27.625926 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:27.625840 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-99nm4_b37139c1-69bb-4dda-87f4-ec27229b72ef/copy-manifests/0.log" Jul 08 18:18:27.632546 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:27.632525 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-99nm4_b37139c1-69bb-4dda-87f4-ec27229b72ef/manager/0.log" Jul 08 18:18:29.213907 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:29.213878 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jjdgg_5b021dc6-60bc-45de-bd13-f7872a635a49/manager/0.log" Jul 08 18:18:29.586000 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:29.585913 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rj656_a47cbe8a-a1b7-41b3-8518-154cd34681f9/registry-server/0.log" Jul 08 18:18:29.719284 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:29.719251 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-kdcvq_85b1610a-4937-4054-9026-c97b70bc9a55/manager/0.log" Jul 08 18:18:29.836278 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:29.836195 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-72t8w_dee51a9c-58e0-4d02-8e8e-74a9beac6c6b/limitador/0.log" Jul 08 18:18:30.554041 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:30.554012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-89v6r_29f5b358-7821-49d3-98a1-52a170ee8960/kube-auth-proxy/0.log" Jul 08 18:18:31.147866 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:31.147834 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59c88ffcb5-g8nl9_e853ca21-cbdf-4916-bd5e-6755ea765ac7/router/0.log" Jul 08 18:18:31.608940 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:31.608909 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-sqrjq_a6bdd59b-06df-4203-b427-17b9d148f688/main/0.log" Jul 08 18:18:31.955076 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:31.955049 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ncpxw_7e550abd-0e78-4483-8584-85b2280c5fd4/main/0.log" Jul 08 18:18:39.379068 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:39.379030 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7x7wz_4a043eb4-3ebd-42f9-b1b9-0d96dd93553a/global-pull-secret-syncer/0.log" Jul 08 18:18:39.603768 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:39.603742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x66br_8a76373e-ad42-4a73-b723-1cb32ed2f2b7/konnectivity-agent/0.log" Jul 08 18:18:39.654343 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:39.654272 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-65.ec2.internal_94467296c0cab4e5954502ea1cc5498e/haproxy/0.log" Jul 08 18:18:44.186109 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:44.186079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jjdgg_5b021dc6-60bc-45de-bd13-f7872a635a49/manager/0.log" Jul 08 18:18:44.301513 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:44.301487 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rj656_a47cbe8a-a1b7-41b3-8518-154cd34681f9/registry-server/0.log" Jul 08 18:18:44.440670 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:44.440642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-kdcvq_85b1610a-4937-4054-9026-c97b70bc9a55/manager/0.log" Jul 08 18:18:44.497938 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:44.497904 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-72t8w_dee51a9c-58e0-4d02-8e8e-74a9beac6c6b/limitador/0.log" Jul 08 18:18:46.687656 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:46.687616 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-gg28g_7d531774-6047-4988-9880-c8d5044bfd0d/cluster-monitoring-operator/0.log" Jul 08 18:18:46.896420 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:46.896387 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2prw7_26313d58-957b-49df-a547-4fc1df155f5d/node-exporter/0.log" Jul 08 18:18:46.928654 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:46.928626 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2prw7_26313d58-957b-49df-a547-4fc1df155f5d/kube-rbac-proxy/0.log" Jul 08 18:18:46.956325 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:46.956304 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2prw7_26313d58-957b-49df-a547-4fc1df155f5d/init-textfile/0.log" Jul 08 18:18:47.750779 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:47.750747 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-47jdl_48f06d89-7333-407d-b4db-d5e5d9355cf6/prometheus-operator-admission-webhook/0.log" Jul 08 18:18:49.237893 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.237868 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-h7whl_019ebb78-1135-4641-a93e-8b1f5e4e4d6b/networking-console-plugin/0.log" Jul 08 18:18:49.850934 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.850900 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j"] Jul 08 18:18:49.851326 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.851312 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" containerName="test-curl-7634-140482028745040" Jul 08 18:18:49.851370 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.851328 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" containerName="test-curl-7634-140482028745040" Jul 08 18:18:49.851409 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.851398 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b18c31-cef4-464d-ae11-38bdcaa2a4e9" containerName="test-curl-7634-140482028745040" Jul 08 18:18:49.854747 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.854726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:49.856942 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.856921 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq44k\"/\"openshift-service-ca.crt\"" Jul 08 18:18:49.857703 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.857686 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq44k\"/\"kube-root-ca.crt\"" Jul 08 18:18:49.857789 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.857685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cq44k\"/\"default-dockercfg-5jf2t\"" Jul 08 18:18:49.863141 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.863120 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j"] Jul 08 18:18:49.889398 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.889368 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/1.log" Jul 08 18:18:49.892936 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:49.892918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wspx5_20bcd1ae-4584-4f6a-a8ec-90982f45475d/console-operator/2.log" Jul 08 18:18:50.005645 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.005619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-podres\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.005809 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.005652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-sys\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.005809 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.005680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-lib-modules\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.005809 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.005756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-proc\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.005964 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.005813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwtdm\" (UniqueName: \"kubernetes.io/projected/302ad526-abd0-48e4-b39c-4866a5c51b10-kube-api-access-vwtdm\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.106997 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.106915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-proc\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.106997 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.106963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwtdm\" (UniqueName: \"kubernetes.io/projected/302ad526-abd0-48e4-b39c-4866a5c51b10-kube-api-access-vwtdm\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-podres\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-proc\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-sys\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-lib-modules\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-sys\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107180 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-podres\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.107212 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.107191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/302ad526-abd0-48e4-b39c-4866a5c51b10-lib-modules\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.115198 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.115172 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwtdm\" (UniqueName: \"kubernetes.io/projected/302ad526-abd0-48e4-b39c-4866a5c51b10-kube-api-access-vwtdm\") pod \"perf-node-gather-daemonset-zw82j\" (UID: \"302ad526-abd0-48e4-b39c-4866a5c51b10\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.165940 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.165915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:50.303875 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.303850 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j"] Jul 08 18:18:50.306530 ip-10-0-129-65 kubenswrapper[2579]: W0708 18:18:50.306501 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod302ad526_abd0_48e4_b39c_4866a5c51b10.slice/crio-70d8c60fd5c9e1648ed9ab758f1737baddfdbaf4789a4abda801857fbf300cec WatchSource:0}: Error finding container 70d8c60fd5c9e1648ed9ab758f1737baddfdbaf4789a4abda801857fbf300cec: Status 404 returned error can't find the container with id 70d8c60fd5c9e1648ed9ab758f1737baddfdbaf4789a4abda801857fbf300cec Jul 08 18:18:50.456488 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:50.456464 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-qhmvk_e9db5d99-bf1f-4a6e-bfe7-e55ed3a41d34/download-server/0.log" Jul 08 18:18:51.017110 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:51.017070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" event={"ID":"302ad526-abd0-48e4-b39c-4866a5c51b10","Type":"ContainerStarted","Data":"542ca2c868ef390059d060f919bb32bc25fe89a3a75e68e19952e3d96007a646"} Jul 08 18:18:51.017309 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:51.017122 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" event={"ID":"302ad526-abd0-48e4-b39c-4866a5c51b10","Type":"ContainerStarted","Data":"70d8c60fd5c9e1648ed9ab758f1737baddfdbaf4789a4abda801857fbf300cec"} Jul 08 18:18:51.017309 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:51.017197 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:51.040545 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:51.040498 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" podStartSLOduration=2.040484355 podStartE2EDuration="2.040484355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:51.037921589 +0000 UTC m=+2733.840008107" watchObservedRunningTime="2026-07-08 18:18:51.040484355 +0000 UTC m=+2733.842570873" Jul 08 18:18:51.045899 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:51.045879 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-pbmdt_defd5d3c-19fc-47a3-9e7e-08a5e04c17e3/volume-data-source-validator/0.log" Jul 08 18:18:52.190174 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:52.190146 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-td4sp_b3651a18-f091-4b86-8653-8b1f07453339/dns/0.log" Jul 08 18:18:52.214892 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:52.214865 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-td4sp_b3651a18-f091-4b86-8653-8b1f07453339/kube-rbac-proxy/0.log" Jul 08 18:18:52.264315 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:52.264291 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kthzc_5997a67d-d6be-44d7-be12-0c42de6ded38/dns-node-resolver/0.log" Jul 08 18:18:52.856332 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:52.856304 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2kz8j_f80c2ab9-7f70-4fb4-a04d-6c865232a4a7/node-ca/0.log" Jul 08 18:18:54.079919 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:54.079892 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-89v6r_29f5b358-7821-49d3-98a1-52a170ee8960/kube-auth-proxy/0.log" Jul 08 18:18:54.300369 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:54.300315 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59c88ffcb5-g8nl9_e853ca21-cbdf-4916-bd5e-6755ea765ac7/router/0.log" Jul 08 18:18:54.996825 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:54.996795 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gmzgc_e5d590a0-e899-485c-901e-088c7bdabc9a/serve-healthcheck-canary/0.log" Jul 08 18:18:55.494631 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:55.494584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cjdr8_c32f2368-e81f-42ea-804f-62c5d495c6a8/insights-operator/0.log" Jul 08 18:18:55.495060 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:55.494963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-cjdr8_c32f2368-e81f-42ea-804f-62c5d495c6a8/insights-operator/1.log" Jul 08 18:18:55.594175 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:55.594150 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tn2xt_3c6223a6-2690-4fae-8e29-692c0118f36e/kube-rbac-proxy/0.log" Jul 08 18:18:55.617393 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:55.617340 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tn2xt_3c6223a6-2690-4fae-8e29-692c0118f36e/exporter/0.log" Jul 08 18:18:55.640062 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:55.640038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tn2xt_3c6223a6-2690-4fae-8e29-692c0118f36e/extractor/0.log" Jul 08 18:18:57.032190 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:57.032162 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-zw82j" Jul 08 18:18:57.809199 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:57.809171 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fwl8z_16affe09-6490-406d-8019-d0e4c7c2bcf4/manager/0.log" Jul 08 18:18:57.940941 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:57.940913 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fjpln_8dbe8b09-e32a-4adf-8dd2-0d53d2fa5dd0/manager/0.log" Jul 08 18:18:58.053090 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:58.053051 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-x4f7b_194fd4db-b502-40ce-b463-9ef5d3339207/server/0.log" Jul 08 18:18:58.115449 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:58.115380 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r5txw_686114d0-cc38-45b7-89f5-ee98b6b42723/manager/0.log" Jul 08 18:18:58.239082 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:58.239057 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-99nm4_b37139c1-69bb-4dda-87f4-ec27229b72ef/manager/0.log" Jul 08 18:18:58.261161 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:18:58.261134 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-99nm4_b37139c1-69bb-4dda-87f4-ec27229b72ef/copy-manifests/0.log" Jul 08 18:19:00.287214 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:00.287170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-89v6r_29f5b358-7821-49d3-98a1-52a170ee8960/kube-auth-proxy/0.log" Jul 08 18:19:00.449430 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:00.449395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59c88ffcb5-g8nl9_e853ca21-cbdf-4916-bd5e-6755ea765ac7/router/0.log" Jul 08 18:19:01.758275 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:01.758249 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-89v6r_29f5b358-7821-49d3-98a1-52a170ee8960/kube-auth-proxy/0.log" Jul 08 18:19:01.936768 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:01.936740 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59c88ffcb5-g8nl9_e853ca21-cbdf-4916-bd5e-6755ea765ac7/router/0.log" Jul 08 18:19:04.913221 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:04.913190 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-8sz8l_3bf6faff-4e8f-436a-92b9-8fb3411f3a06/migrator/0.log" Jul 08 18:19:04.947738 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:04.947716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-8sz8l_3bf6faff-4e8f-436a-92b9-8fb3411f3a06/graceful-termination/0.log" Jul 08 18:19:05.323791 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:05.323762 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-qrdlt_d0c4509a-bf4d-423b-82d1-72d7f666a32f/kube-storage-version-migrator-operator/1.log" Jul 08 18:19:05.324375 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:05.324359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-qrdlt_d0c4509a-bf4d-423b-82d1-72d7f666a32f/kube-storage-version-migrator-operator/0.log" Jul 08 18:19:06.766094 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.766067 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/kube-multus-additional-cni-plugins/0.log" Jul 08 18:19:06.787311 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.787281 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/egress-router-binary-copy/0.log" Jul 08 18:19:06.808535 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.808512 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/cni-plugins/0.log" Jul 08 18:19:06.830098 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.830074 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/bond-cni-plugin/0.log" Jul 08 18:19:06.851485 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.851462 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/routeoverride-cni/0.log" Jul 08 18:19:06.873741 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.873725 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/whereabouts-cni-bincopy/0.log" Jul 08 18:19:06.896138 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.896114 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qk9v6_2170557d-59b8-4c8a-8575-d9ca35586250/whereabouts-cni/0.log" Jul 08 18:19:06.997863 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:06.997836 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pk4h7_fa2450d5-725a-40c0-9c41-0ea21e082328/kube-multus/0.log" Jul 08 18:19:07.120012 ip-10-0-129-65 kubenswrapper[2579]: I0708 18:19:07.119937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmqqm_d1bc455f-6a8a-4c83-aef5-022768f814f0/network-metrics-daemon/0.log"